Databricks Data Pipelines Engineer II – Spark & Delta Lake

JPMorgan Chase & Co.; Bournemouth; On-site; GBP 70,000 - 110,000; Full time

Job summary

JPMorgan Chase & Co. in the United Kingdom seeks a Databricks Engineer to design, build, and operate scalable data pipelines on the Databricks Lakehouse platform.

You will partner with data analysts, data scientists, and application teams to deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products. The role emphasizes batch/streaming pipelines, Spark/PySpark, Delta Lake, governance and security, with CI/CD and collaboration across environments.

Qualifications

• Experience with building data pipelines on Databricks and/or Apache Spark in production.
• Strong coding skills in Python (PySpark) and SQL (Scala a plus).
• Hands-on experience with Delta Lake features (MERGE/UPSERT, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM).
• Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data Factory, etc.).
• Familiarity with cloud data platforms (AWS/Azure/GCP) and storage (S3/ADLS/GCS).
• Solid understanding of data engineering fundamentals: data modeling, ETL/ELT patterns, reliability, observability, and performance tuning.
• Hands-on experience using enterprise-authorized AI-assisted software development tools with demonstrated ability to evaluate AI outputs.
• Understanding of responsible AI use in engineering workflows, including data sensitivity considerations and secure handling of inputs/outputs

Responsibilities

• Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators (e.g., Workflows, Airflow, ADF).
• Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency.
• Build and maintain curated data models (bronze/silver/gold), data quality checks, and automated testing.
• Implement CI/CD for notebooks and code (Git-based workflows), and automate deployments across environments.
• Manage and tune Databricks clusters, jobs, and configurations; monitor production workloads and resolve incidents.
• Integrate multiple data sources (cloud storage, relational DBs, APIs, event streams) and implement robust ingestion patterns.
• Apply data governance and security best practices (access controls, secrets management, lineage/metadata, auditing).
• Create clear documentation for pipelines, data contracts, and operational runbooks.
• Leverages enterprise-authorized AI coding assist tools to improve code quality, delivery speed, and productivity, while validating outputs through peer review and testing.
• Applies knowledge of SDLC tools and automation capabilities to improve value realized by automation

Skills

Python (PySpark)

SQL

Spark

Delta Lake

Databricks

Airflow

CI/CD

Data Modeling

Data Governance

Security Best Practices

Tools

Databricks (Spark)

Delta Lake

Airflow

Azure Data Factory

S3/ADLS/GCS

Git-based Workflows

Job description

JPMorgan Chase & Co. in the United Kingdom seeks a Databricks Engineer to design, build, and operate scalable data pipelines on the Databricks Lakehouse platform.

You will partner with data analysts, data scientists, and application teams to deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products. The role emphasizes batch/streaming pipelines, Spark/PySpark, Delta Lake, governance and security, with CI/CD and collaboration across environments.