Lead Data Engineer: Python, Databricks & AWS

Next Frontier Capital

Glasgow

On-site

GBP 90,000 - 120,000

Full time

14 days+

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

JPMorgan Chase in the United Kingdom is seeking a Lead Software Engineer to design, develop, and maintain scalable data processing solutions using Databricks, Python, and AWS. You will collaborate with cross-functional teams to turn data into actionable insights that support strategic decisions.

You will mentor associates, guide SDLC, ensure data quality and security, implement monitoring, and drive modernization of data warehouses.

Qualifications

Proven experience in data management, ETL/ELT pipelines, and large-scale data processing.
Proficiency in SQL, Python, and PySpark, with query optimization and performance tuning.
Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).
Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).
Strong understanding of data quality, security, and lineage best practices.
Experience with cloud-based data warehouse migration and modernization.
Proficient in CI/CD, delivery methods, the SDLC, and Agile methodologies.
In-depth knowledge of the financial services industry and their IT systems.
Excellent problem-solving, troubleshooting, and analytical skills.
Strong communication and documentation abilities.

Responsibilities

Execute creative, data-driven software solutions, including design, development, and troubleshooting, with the ability to think beyond routine approaches.
Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, and Databricks on AWS.
Develop fact and dimension data models for reporting and analytics.
Write secure, high-quality production code, and review code written by others.
Ensure data quality, security, and lineage throughout data processing and transformation; implement monitoring and alerting.
Support data migration and modernization initiatives, transitioning legacy systems to cloud-based data warehouses.
Lead communities of practice across Software Engineering to promote adoption of new technologies; foster diversity and inclusion.
Mentor Associate-level engineers, supporting their development and delivery standards.
Collaborate with stakeholders to transform data into insights for strategic decisions.
Document data flows, logic, and transformation rules for transparency and knowledge sharing.
Lead and participate in the full SDLC, including requirements, design, development, testing, deployment, and maintenance.
Identify and automate remediation of recurring issues to improve operational stability; act as Production Support and SRE for pipelines and services.
Utilise AI tools to accelerate development and testing of data pipelines.