Lead Data Engineer: Cloud-Native Analytics & Platform Leadership

慨正橡扯; Greater London; On-site; GBP 110,000 - 150,000; Full time

Job summary

J.P. Morgan is seeking a Lead Data Engineer within Personal Investing to design, build, and operate a cloud-native data platform powering analytics, regulatory reporting, and data-driven products for UK investors.

You will mentor engineers, set technical direction, and collaborate with product and analytics teams while applying modern data engineering patterns and strong software fundamentals. This role emphasizes ownership, observability, security, and scalable pipelines using Python, PySpark,

Qualifications

• Degree in Computer Science or a STEM field (or equivalent).
• Demonstrated experience delivering in an agile, fast-paced engineering environment.
• 8 years of hands-on experience coding as a data engineer.
• Strong software engineering fundamentals and testing strategies.
• Cloud-based data platforms using major cloud services (AWS, Google Cloud, or Azure).
• Experience with large-scale distributed data processing and performance tuning.
• Experience with data warehousing/lakehouse technologies and engines (e.g., Spark, Trino).
• Strong SQL skills and dbt for transformations.
• Experience designing and operating orchestration pipelines (Airflow).
• Experience building streaming pipelines (Kafka, Pub/Sub).

Responsibilities

• Design scalable, reusable data processing frameworks using Python, PySpark, dbt.
• Build and optimize batch and streaming pipelines with high performance and observability.
• Develop and operate orchestration with Airflow to schedule and monitor data movement.
• Model and transform data for analytics using SQL and dbt.
• Write production-grade Python/PySpark code with testing and OOD design.
• Implement infrastructure-as-code (Terraform) to provision cloud components.
• Containerize and deploy services with Docker and Kubernetes.
• Collaborate with analysts, data scientists, and application teams to translate requirements into designs.
• Own critical data systems to improve reliability, security, and operations.
• Mentor junior engineers and influence technical direction through standards.

Skills

Python

PySpark

Software engineering

Agile methodologies

SQL

Testing

OO design

Cloud data platforms

Education

Bachelor's degree in Computer Science or STEM-related field

Tools

Airflow

dbt

Terraform

Docker

Kubernetes

Spark

Flink

Trino

Iceberg

Hudi

Redshift

BigQuery

Snowflake

Job description

J.P. Morgan is seeking a Lead Data Engineer within Personal Investing to design, build, and operate a cloud-native data platform powering analytics, regulatory reporting, and data-driven products for UK investors.

You will mentor engineers, set technical direction, and collaborate with product and analytics teams while applying modern data engineering patterns and strong software fundamentals. This role emphasizes ownership, observability, security, and scalable pipelines using Python, PySpark,