Databricks Data Engineer II: Scalable Pipelines

JPMorganChase; Bournemouth; On-site; GBP 90,000 - 120,000; Full time

Job summary

JPMorgan Chase is seeking a Software Engineer II in the Corporate & Investment Bank Payments Technology team to design, build, and operate scalable data pipelines on the Databricks Lakehouse platform.

We’re looking for a Databricks Engineer to design, build, and operate scalable data pipelines and analytics workloads, partnering with data analysts and scientists to deliver trusted datasets and performant ETL/ELT pipelines.

Qualifications

• Experience with building data pipelines on Databricks and/or Apache Spark in production.
• Strong coding skills in Python (PySpark) and SQL.
• Hands-on experience with Delta Lake features (MERGE/UPSERT, schema evolution, partitioning).
• Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data Factory).
• Familiarity with cloud data platforms (AWS/Azure/GCP) and storage (S3/ADLS/GCS).
• Understanding of data governance, security, lineage, and auditing.

Responsibilities

• Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators.
• Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance and cost efficiency.
• Build and maintain curated data models (bronze/silver/gold) with data quality checks and automated testing.
• Implement CI/CD for notebooks and code, and automate deployments across environments.
• Manage and tune Databricks clusters, jobs, and configurations; monitor production workloads.
• Integrate multiple data sources and ensure robust ingestion patterns with governance.
• Create documentation for pipelines, data contracts, and runbooks.
• Leverage AI-assisted development tools with careful validation and secure coding.

Skills

Databricks

Spark (PySpark)

Python

SQL

Delta Lake

Airflow

Azure Data Factory

Cloud platforms

Data governance

Tools

Databricks

Airflow

Azure Data Factory

Delta Lake

Unity Catalog

Job description

JPMorgan Chase is seeking a Software Engineer II in the Corporate & Investment Bank Payments Technology team to design, build, and operate scalable data pipelines on the Databricks Lakehouse platform.

We’re looking for a Databricks Engineer to design, build, and operate scalable data pipelines and analytics workloads, partnering with data analysts and scientists to deliver trusted datasets and performant ETL/ELT pipelines.