Lead Data Engineer

London Stock Exchange Group; Hyderabad; Hybrid; INR 1,800,000 - 2,800,000; Full time

Benefits offered by this job

Hybrid work arrangement

Job summary

London Stock Exchange Group (LSEG) seeks a Lead Data Engineer for our billing transformation program in India. You will source telemetry data, build scalable pipelines in Snowflake/AWS Glue, implement rating logic, and deliver invoice-ready outputs for SAP invoicing. Expect automation, data quality, and robust monitoring in a hybrid work setup.

The role emphasizes CI/CD, data lineage, and collaboration with Finance, Engineering, and Analytics to ensure billing accuracy and process improvements.

Qualifications

• Experience in data engineering, ETL/ELT development, and large-scale data pipeline design.
• Experience with AWS Glue, PySpark, Python, SQL, and cloud-based data processing.
• Strong Snowflake experience as a data warehouse or enterprise data platform.
• Knowledge of usage-based billing, rating engines, or financial data processing.
• Experience with data quality checks, reconciliation frameworks, audit controls, and exception handling.
• Experience with CI/CD tools like GitHub Actions, GitLab CI, Jenkins, AWS Code Pipeline.
• Strong SQL for data analysis, validation, troubleshooting, and performance tuning.
• Experience with batch and event-driven data ingestion.

Responsibilities

• Data ingestion pipelines from telemetry systems.
• Curated usage datasets in Snowflake.
• Rating logic implementation and billing-ready datasets.
• SAP invoice feed generation.
• Data quality and reconciliation framework.
• Exception and error-handling process.
• CI/CD pipelines for data engineering deployments.
• Monitoring, alerting, and operational dashboards.
• Technical documentation and runbooks.

Skills

ETL/ELT development

Large-scale data pipelines

Strong SQL

Batch & event-driven ingestion

Data modelling & performance tuning

CI/CD practices

Tools

AWS Glue

Snowflake

PySpark

Python

SQL

GitHub Actions

GitLab CI

Jenkins

AWS CodePipeline

Job description

Role Overview

We are seeking an experienced Lead Data Engineer to join our billing transformation program. The engineer will be responsible for sourcing telemetry data, building scalable data pipelines, supporting rating logic, and preparing billing-ready outputs for SAP invoicing. The role requires strong experience with AWS Glue, Snowflake, Python, SQL, and CI/CD practices. The candidate will also be responsible for implementing data quality checks, reconciliation controls, audit trails, and operational monitoring to ensure billing accuracy. This is a critical role supporting a transformation from raw telemetry usage data to rated, validated, and invoice-ready billing data.

Key Responsibilities

• Data ingestion pipelines from telemetry systems.
• Curated usage datasets in Snowflake.
• Rating logic implementation and billing-ready datasets.
• SAP invoice feed generation.
• Data quality and reconciliation framework.
• Exception and error-handling process.
• CI/CD pipelines for data engineering deployments.
• Monitoring, alerting, and operational dashboards.
• Technical documentation and runbooks.

Required Skills

• Strong hands-on experience in data engineering, ETL/ELT development, and large-scale data pipeline design.
• Experience with AWS Glue, PySpark, Python, SQL, and cloud-based data processing.
• Strong experience working with Snowflake as a data warehouse or enterprise data platform.
• Good understanding of usage-based billing, rating engines, invoice feeds, or financial data processing.
• Experience building data quality checks, reconciliation frameworks, audit controls, and exception handling processes.
• Hands-on experience with CI/CD tools such as GitHub Actions, GitLab CI, Jenkins, AWS Code Pipeline, or similar.
• Strong SQL skills for data analysis, validation, troubleshooting, and performance optimization.
• Experience working with batch and event-driven data ingestion patterns.
• Understanding of data modelling, partitioning, performance tuning, and cost optimization.

Soft Skills

• Strong analytical skills and attention to detail—especially important for financial accuracy.
• Ability to collaborate with cross-functional teams including Finance, Engineering, and Analytics.
• Good communication skills to explain defects, logic, and test outcomes clearly.
• High ownership and adaptability, with willingness to take on both QA and minor development tasks.

Benefits

• We are an equal opportunity employer with hybrid working.
• Annual Leave and Parental Leave benefits.
• Upskilling opportunities with job-relevant training and certifications.
• Dedicated Employee Assistance and Wellbeing support.

What you’ll get in return

• High-impact projects: We work on a variety of pioneering AI products and leverage extensive datasets to solve complex, high-value challenges.
• Competitive benefits: You will enjoy a strong compensation package, comprehensive benefits, and ongoing investment in career growth and skill development.
• Industry leadership: This is an opportunity to be a founding member of the organization that’s delivering brand-new products that democratize modeling and analytics solutions.
• Collaborative environment: We provide opportunities for continuous learning and professional development in a work environment of dedicated, highly experienced teams. We recognize that to attract the best talent, we need to be flexible, and we are open to discussing work arrangements with you. We take a hybrid approach to the workplace; this role is considered ‘Blended’, which requires attending the office at least three day per week while some teams and colleagues choose to collaborate in the office more frequently. We're proud to have been recognised as a Great Place to Work® in India ‘25 Learn more about life and purpose of our company directly from India colleagues’ video: Bengaluru, India | Where We Work | LSEG Career Stage: Senior Associate London Stock Exchange Group (LSEG) Information: Join us and be part of a team that values innovation, quality, and continuous improvement. If you're ready to take your career to the next level and make a significant impact, we'd love to hear from you. LSEG is a leading global financial markets infrastructure and data provider. Our purpose is driving financial stability, empowering economies and enabling customers to create sustainable growth. Our purpose is the foundation on which our culture is built. Our values of Integrity, Partnership, Excellence and Change underpin our purpose and set the standard for everything we do, every day. They go to the heart of who we are and guide our decision making and everyday actions. Working with us means that you will be part of a dynamic organisation of 25,000 people across 65 countries. However, we will value your individuality and enable you to bring your true self to work so you can help enrich our diverse workforce. We are proud to be an equal opportunities employer. This means that we do not discriminate on the basis of anyone’s race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected under applicable law. Conforming with applicable law, we can reasonably accommodate applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs. You will be part of a collaborative and creative culture where we encourage new ideas. We are committed to sustainability across our global business and we are proud to partner with our customers to help them meet their sustainability objectives. Our charity, the LSEG Foundation provides charitable grants to community groups that help people access economic opportunities and build a secure future with financial independence. Colleagues can get involved through fundraising and volunteering. LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days and wellbeing initiatives.

Please take a moment to read this privacy notice carefully, as it describes what personal information London Stock Exchange Group (LSEG) (we) may hold about you, what it’s used for, and how it’s obtained, your rights and how to contact us as a data subject.