Lead Data Engineer

LSEG; Hyderabad; Hybrid; INR 3,000,000 - 6,000,000; Full time

Benefits offered by this job

Hybrid working

Leave benefits

Upskilling opportunities

Wellbeing support

Job summary

LSEG seeks an experienced Lead Data Engineer for its billing transformation program in Hyderabad. You will source telemetry, build scalable pipelines, and support rating logic to deliver invoice-ready data for SAP invoicing.

Role emphasizes data quality, reconciliation controls, and CI/CD practices, with a strong focus on monitoring and operational excellence.

Qualifications

• Experience in data engineering and large-scale pipeline design.
• Hands-on with AWS Glue, PySpark, Python and SQL in cloud data processing.
• Strong Snowflake experience as a data warehouse/enterprise platform.
• Understanding of billing data processing and usage-based invoicing is a plus.
• Experience building data quality checks, reconciliation tools and audit controls.
• Proficient in CI/CD tools for deployment automation.

Responsibilities

• Ingest telemetry data and build scalable data pipelines.
• Create curated usage datasets in Snowflake and implement rating logic.
• Generate SAP invoice feeds and ensure billing-ready data quality.
• Develop CI/CD pipelines and maintain runbooks and dashboards.
• Monitor data pipelines and implement reconciliation and audit controls.

Skills

Data engineering

ETL/ELT design

AWS Glue

PySpark

Python

SQL

CI/CD

Data quality checks

Reconciliation frameworks

Audit controls

GitHub Actions

Snowflake

Batch & event-driven ingestion

Performance tuning

Tools

AWS Glue

Snowflake

Python

SQL

GitHub Actions

GitLab CI

Jenkins

AWS Code Pipeline

Job description

Role Overview

We are seeking an experienced Lead Data Engineer to join our billing transformation program. The engineer will be responsible for sourcing telemetry data, building scalable data pipelines, supporting rating logic, and preparing billing-ready outputs for SAP invoicing.

Role Overview

We are seeking an experienced Lead Data Engineer to join our billing transformation program. The engineer will be responsible for sourcing telemetry data, building scalable data pipelines, supporting rating logic, and preparing billing-ready outputs for SAP invoicing.

The role requires strong experience with AWS Glue, Snowflake, Python, SQL, and CI/CD practices. The candidate will also be responsible for implementing data quality checks, reconciliation controls, audit trails, and operational monitoring to ensure billing accuracy.

This is a critical role supporting a transformation from raw telemetry usage data to rated, validated, and invoice-ready billing data.

Key Responsibilities

• Data ingestion pipelines from telemetry systems.
• Curated usage datasets in Snowflake.
• Rating logic implementation and billing-ready datasets.
• SAP invoice feed generation.
• Data quality and reconciliation framework.
• Exception and error-handling process.
• CI/CD pipelines for data engineering deployments.
• Monitoring, alerting, and operational dashboards.
• Technical documentation and runbooks.

Required Skills

• Strong hands-on experience in data engineering, ETL/ELT development, and large-scale data pipeline design.
• Experience with AWS Glue, PySpark, Python, SQL, and cloud-based data processing.
• Strong experience working with Snowflake as a data warehouse or enterprise data platform.
• Good understanding of usage-based billing, rating engines, invoice feeds, or financial data processing.
• Experience building data quality checks, reconciliation frameworks, audit controls, and exception handling processes.
• Hands-on experience with CI/CD tools such as GitHub Actions, GitLab CI, Jenkins, AWS Code Pipeline, or similar.
• Strong SQL skills for data analysis, validation, troubleshooting, and performance optimization.
• Experience working with batch and event-driven data ingestion patterns.
• Understanding of data modelling, partitioning, performance tuning, and cost optimization.

Soft Skills

• Strong analytical skills and attention to detail—especially important for financial accuracy.
• Ability to collaborate with cross-functional teams including Finance, Engineering, and Analytics.
• Good communication skills to explain defects, logic, and test outcomes clearly.
• High ownership and adaptability, with willingness to take on both QA and minor development tasks.

Benefits

• We are an equal opportunity employer with hybrid working.
• Annual Leave and Parental Leave benefits.
• Upskilling opportunities with job-relevant training and certificates.
• Dedicated Employee Assistance and Wellbeing support.

What You’ll Get In Return

High-impact projects: We work on a variety of pioneering AI products and leverage extensive datasets to solve complex, high-value challenges.
Competitive benefits: You will enjoy a strong compensation package, comprehensive benefits, and ongoing investment in career growth and skill development.
Industry leadership: This is an opportunity to be a founding member of the organization that’s delivering brand-new products that democratize modeling and analytics solutions.
Collaborative environment: We provide opportunities for continuous learning and professional development in a work environment of dedicated, highly experienced teams.

We recognize that to attract the best talent, we need to be flexible, and we are open to discussing work arrangements with you. We take a hybrid approach to the workplace; this role is considered ‘Blended’, which requires attending the office at least three day per week while some teams and colleagues choose to collaborate in the office more frequently.

We're proud to have been recognised as a Great Place to Work in India ‘25

Learn more about life and purpose of our company directly from India colleagues’ video: Bengaluru, India | Where We Work | LSEG

Career Stage

Senior Associate

London Stock Exchange Group (LSEG) Information

Join us and be part of a team that values innovation, quality, and continuous improvement. If you're ready to take your career to the next level and make a significant impact, we'd love to hear from you.

LSEG is a leading global financial markets infrastructure and data provider. Our purpose is driving financial stability, empowering economies and enabling customers to create sustainable growth.

Our purpose is the foundation on which our culture is built. Our values of Integrity, Partnership, Excellence and Change underpin our purpose and set the standard for everything we do, every day. They go to the heart of who we are and guide our decision making and everyday actions.

Working with us means that you will be part of a dynamic organisation of 25,000 people across 65 countries. However, we will value your individuality and enable you to bring your true self to work so you can help enrich our diverse workforce.

We are proud to be an equal opportunities employer. This means that we do not discriminate on the basis of anyone’s race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy or disability, or any other basis protected under applicable law. Conforming with applicable law, we can reasonably accommodate applicants' and employees' religious practices and beliefs, as well as mental health or physical disability needs.

You will be part of a collaborative and creative culture where we encourage new ideas. We are committed to sustainability across our global business and we are proud to partner with our customers to help them meet their sustainability objectives. Our charity, the LSEG Foundation provides charitable grants to community groups that help people access economic opportunities and build a secure future with financial independence. Colleagues can get involved through fundraising and volunteering.

LSEG offers a range of tailored benefits and support, including healthcare, retirement planning, paid volunteering days and wellbeing initiatives.

Please take a moment to read this privacy notice carefully, as it describes what personal information London Stock Exchange Group (LSEG) (we) may hold about you, what it’s used for, and how it’s obtained, your rights and how to contact us as a data subject.

If you are submitting as a Recruitment Agency Partner, it is essential and your responsibility to ensure that candidates applying to LSEG are aware of this privacy notice.