Software Engineer II - Databricks

JPMorgan Chase & Co.; Christchurch; On-site; GBP 75,000 - 110,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Software Engineer II in the Payments Technology team to design, build, and operate scalable data pipelines on the Databricks Lakehouse.

You will collaborate with data analysts, data scientists, and application teams to deliver trusted datasets and well-governed data products. Responsibilities include designing batch/streaming pipelines with Databricks Spark, Delta Lake, and orchestrators; developing Spark jobs and SQL transformations; building data models and

Qualifications

• Experience building production data pipelines on Databricks and/or Apache Spark.
• Strong coding skills in Python (PySpark) and SQL.
• Hands-on Delta Lake usage including MERGE/UPSERT, schema evolution, partitioning, and OPTIMIZE/VACUUM.
• Experience with orchestration/scheduling (Databricks Workflows, Airflow, Azure Data Factory).
• Familiarity with cloud data platforms (AWS/Azure/GCP) and storage (S3/ADLS/GCS).
• Solid understanding of data engineering fundamentals: data modeling, ETL/ELT, reliability, observability, performance tuning.
• Experience with enterprise-authorized AI-assisted development tools and validating AI outputs.

Responsibilities

• Design and implement batch and streaming data pipelines on Databricks (Spark) and Delta Lake.
• Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance and cost efficiency.
• Build and maintain curated data models and data quality checks.
• Implement CI/CD for notebooks and code and automate deployments across environments.
• Manage and tune Databricks clusters, jobs, and configurations; monitor production workloads.
• Integrate multiple data sources and ensure robust ingestion patterns.
• Apply data governance, security practices, and lineage/metadata management.
• Create documentation for pipelines and data contracts.

Skills

Databricks

Python (PySpark)

SQL

Data modeling

ETL/ELT

CI/CD

Data governance

Tools

Delta Lake

Databricks Workflows

Airflow

Azure Data Factory

S3/ADLS/GCS

Spark

Job description

As a Software Engineer II at JPMorgan Chase within our Corporate Investment Bank , Payments Technology team, you'll design, build, and operate scalable data pipelines and analytics workloads on the Databricks Lakehouse platform. We're looking for a Databricks Engineer to design, build, and operate scalable data pipelines and analytics workloads on the Databricks Lakehouse platform. You'll partner with data analysts, data scientists, and application teams to deliver trusted datasets, performant ETL/ELT pipelines, and well-governed data products.

Job responsibilities

• Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake, and orchestrators (e.g., Workflows, Airflow, ADF).
• Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability, and cost efficiency.
• Build and maintain curated data models (bronze/silver/gold), data quality checks, and automated testing.
• Implement CI/CD for notebooks and code (Git-based workflows), and automate deployments across environments.
• Manage and tune Databricks clusters, jobs, and configurations; monitor production workloads and resolve incidents.
• Integrate multiple data sources (cloud storage, relational DBs, APIs, event streams) and implement robust ingestion patterns.
• Apply data governance and security best practices (access controls, secrets management, lineage/metadata, auditing).
• Create clear documentation for pipelines, data contracts, and operational runbooks.
• Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation

Required qualifications, skills and capabilities

• Experience with building data pipelines on Databricks and/or Apache Spark in production.
• Strong coding skills in Python (PySpark) and SQL (Scala a plus).
• Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM).
• Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure Data Factory, etc.).
• Familiarity with cloud data platforms (AWS/Azure/GCP) and storage (S3/ADLS/GCS).
• Solid understanding of data engineering fundamentals: data modeling, ETL/ELT patterns, reliability, observability, and performance tuning.
• Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, testing, troubleshooting, or documentation) with demonstrated ability to critically evaluate and validate AI-generated outputs.
• Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations

Preferred qualifications

• Experience with streaming (Structured Streaming, Kafka/Event Hubs/Kinesis).
• Experience implementing data quality frameworks (Great Expectations, DQ) and data testing in CI.
• Exposure to Unity Catalog (or similar) for governance and fine-grained permissions.