Lead Data Engineer - London

JP Morgan Chase; Greater London; On-site; GBP 62,000 - 102,000; Full time

Job summary

JP Morgan Chase in London is seeking a Lead Data Engineer to design and scale cloud-native data platforms. You will own data pipelines, collaborate with analysts and data scientists, and drive best practices in reliability, security, and performance.

The role requires 8+ years coding as a data engineer, strong Python, SQL, and experience with AWS/GCP/Azure, Redshift, Snowflake, Spark, Airflow, and Kubernetes. You will mentor juniors and help shape platform standards across the team.

Qualifications

• Degree in Computer Science or STEM, or equivalent
• 8+ years hands-on data engineer experience
• Strong Python, unit/integration testing
• Experience delivering in agile, fast-paced environments
• Cloud data platforms with AWS/GCP/Azure
• Large-scale distributed data processing expertise
• Experience with Redshift, BigQuery or Snowflake
• Spark, Flink or Trino; Iceberg/Hudi or similar
• SQL skills and dbt
• Airflow or similar orchestration
• Kafka or similar messaging
• AI capabilities usage with validation and data sensitivity awareness
• Data modeling for analytics
• Security, risk, governance for data platforms
• CI/CD for data/platform services
• Docker/Kubernetes

Responsibilities

• Design scalable data processing and data quality frameworks with Python, PySpark, and dbt
• Build and optimize batch and streaming pipelines with performance and observability
• Operate Apache Airflow workflows for data movement and transformations
• Model and transform data for analytics using SQL and dbt
• Write production-grade Python and PySpark with tests and OO design
• Implement Terraform to provision cloud components
• Containerize and deploy with Docker/Kubernetes and Helm
• Collaborate with analysts and data scientists to translate requirements into designs
• Own critical data systems, improving reliability, scalability, security
• Mentor junior engineers and shape technical direction
• Use enterprise AI capabilities to accelerate data platform work while ensuring data sensitivity

Skills

Python

SQL

Unit testing

Agile

Data modeling

Cloud data platforms

System design

Hands-on coding

Education

Degree in Computer Science or STEM

Tools

AWS

Google Cloud

Azure

Redshift

BigQuery

Snowflake

Spark

Flink

Trino

Iceberg

Hudi

dbt

Airflow

Kafka

Docker

Kubernetes

Terraform

Job description

Salary: £62,000 - 102,000 per year

Requirements:

• Degree in Computer Science or a STEM-related field, or equivalent


• Demonstrated experience delivering in an agile, fast-paced engineering environment


• 8 years of recent, hands-on professional experience actively coding as a data engineer


• Strong software engineering fundamentals, including system design, data structures, object-oriented programming, testing strategies, and end-to-end development lifecycle


• Strong Python programming skills, including unit and integration testing


• Hands-on experience building and operating cloud-based data platforms using major cloud services such as AWS, Google Cloud, or Azure


• Experience with large-scale distributed data processing and performance tuning


• Hands-on experience with modern data warehousing and lakehouse technologies such as Redshift, BigQuery, or Snowflake, plus engines such as Spark, Flink, or Trino, and table formats such as Iceberg, Hudi, or similar


• Strong SQL skills and experience with SQL-based transformation tooling such as dbt


• Experience designing and operating orchestration pipelines using Airflow or similar tools


• Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems


• Demonstrated experience using enterprise-authorized AI capabilities within the work environment to support data engineering workflows with strong validation habits and awareness of data sensitivity


• Ability to review and validate AI-assisted outputs before use, escalating when uncertain and following data handling requirements


• Data modeling experience for analytics and reporting use cases


• Knowledge of security, risk, compliance, and governance considerations for data platforms


• Experience building continuous integration and continuous delivery automation for data and platform services


• Experience with container-based deployment environments such as Docker and Kubernetes


• Demonstrated ability to coach teammates on engineering practices and contribute to a collaborative, inclusive team culture

Responsibilities:

• Design scalable, reusable data processing and data quality frameworks using Python, PySpark, and dbt


• Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability


• Develop and operate workflow orchestration such as Apache Airflow to schedule, monitor, and manage data movement and transformations


• Model and transform data for analytics using SQL and dbt to support business intelligence and reporting workloads


• Write production-grade Python and PySpark code with disciplined testing, performance tuning, and maintainable object-oriented design


• Implement infrastructure as code such as Terraform to provision and manage cloud-based data platform components


• Containerize and deploy services using Docker and Kubernetes, and related tooling such as Helm


• Collaborate with analysts, data scientists, and application teams to turn requirements into technical designs and delivered solutions


• Own critical data systems by improving reliability, scalability, security, and operational excellence


• Mentor junior engineers and influence the teams technical direction through standards, reviews, and knowledge sharing


• Use enterprise-authorized AI capabilities within the work environment to accelerate data platform and model design analysis and documentation, validating outputs and handling data according to sensitivity and security requirements


• Apply reuse-first, AI-assisted practices within delivery and operational routines, ensuring traceability, auditability, and alignment to resiliency and security expectations

Technologies:

• AI


• Airflow


• AWS


• Redshift


• Azure


• BigQuery


• Business Intelligence


• Cloud


• Docker


• Flink


• Helm


• Support


• Kafka


• Kubernetes


• Python


• PySpark


• SQL


• Security


• Snowflake


• Spark


• Terraform


• dbt


• Marketing

More:

We are J.P. Morgan, a global leader in financial services, providing strategic advice and products to prominent corporations, governments, wealthy individuals, and institutional investors. Within Personal Investing, we are building a modern, cloud-native data platform that supports analytics, regulatory reporting, and data-driven products at scale. This full-time Lead Data Engineer role offers meaningful scope to shape platform standards, mentor others, and grow technical and leadership impact while working in a collaborative environment that values ownership, continuous improvement, diversity, and inclusion.

last updated 37 week of 2026