Lead AWS Software Data Engineer

LSEG; Bengaluru; On-site; INR 3,000,000 - 4,200,000; Full time

Job summary

LSEG – FTSE Russell is seeking a Lead Software Data Engineer to design and implement scalable data pipelines in a modern AWS environment. You will collaborate with product and business teams to deliver reliable, high-quality data solutions powering financial indices globally.

Responsibilities include building robust pipelines, implementing quality checks, and contributing to a lakehouse using Apache Iceberg.

Qualifications

• Write clean, maintainable Python code with types, linters, and tests (pytest).
• Understand data engineering fundamentals: batch processing, schema evolution, ETL pipelines.
• Experience with or eagerness to learn Apache Spark for large-scale data processing.
• Familiarity with the AWS data stack (S3, Glue, Lambda, EMR).
• Strong experience with relational databases (Aurora PostgreSQL).
• Enjoy learning the business context and collaborating with stakeholders.
• Work well in Agile teams and value collaboration over solo heroics.

Responsibilities

• Design and develop scalable, testable data pipelines using Python and Apache Spark.
• Orchestrate data workflows with AWS tools such as Glue, EMR Serverless, Lambda, and S3.
• Apply modern software engineering practices: version control, CI/CD, modular design, and automated testing.
• Contribute to the development of a lakehouse architecture using Apache Iceberg.
• Collaborate with business teams to translate requirements into data-driven solutions.
• Build observability into data flows and implement basic quality checks.
• Participate in code reviews, pair programming, and architecture discussions.
• Continuously learn about the financial indices domain and share insights with the team.

Skills

Python

Apache Spark

AWS data stack

PostgreSQL/Aurora

Agile teamwork

Testing with pytest

Tools

Apache Iceberg

GitLab CI

Jenkins

GitHub Actions

Great Expectations

Deequ

Databricks

Job description

Role Summary

Lead Software Data Engineer – FTSE Russell
Build scalable data systems that power the indices trusted by financial institutions worldwide. Work in a modern AWS environment, develop robust pipelines, and collaborate closely with product and business teams to deliver reliable, high‑quality data solutions.

Responsibilities

• Design and develop scalable, testable data pipelines using Python and Apache Spark.
• Orchestrate data workflows with AWS tools such as Glue, EMR Serverless, Lambda, and S3.
• Apply modern software engineering practices: version control, CI / CD, modular design, and automated testing.
• Contribute to the development of a lakehouse architecture using Apache Iceberg.
• Collaborate with business teams to translate requirements into data‑driven solutions.
• Build observability into data flows and implement basic quality checks.
• Participate in code reviews, pair programming, and architecture discussions.
• Continuously learn about the financial indices domain and share insights with the team.

Qualifications

• Write clean, maintainable, testable Python code (ideally with type hints, linters, and tests such as pytest).
• Understand data engineering fundamentals: batch processing, schema evolution, and building ETL pipelines.
• Experience with or eagerness to learn Apache Spark for large‑scale data processing.
• Familiarity with the AWS data stack (e.g., S3, Glue, Lambda, EMR).
• Strong experience with relational databases (e.g., Aurora PostgreSQL).
• Enjoy learning the business context and working closely with stakeholders.
• Work well in Agile teams and value collaboration over solo heroics.

Nice to Have

• Experience with Apache Iceberg or similar table formats.
• Familiarity with CI / CD tools such as GitLab CI, Jenkins, or GitHub Actions.
• Exposure to data quality frameworks like Great Expectations or Deequ.
• Experience with other data platforms such as Databricks.
• Curiosity about financial markets, index data, or investment analytics.

Equal Opportunity Employer

We are a proud equal‑opportunity employer. We do not discriminate on the basis of race, religion, colour, national origin, gender, sexual orientation, gender identity, gender expression, age, marital status, veteran status, pregnancy, disability, or any other basis protected under applicable law. We provide reasonable accommodations for religious practices and beliefs, and for mental health or physical disability needs.