Lead Databricks Engineer: Data Pipelines & Lakehouse

JPMorgan Chase & Co.; Wilmington (DE); On-site; USD 140,000 - 210,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Lead Software Engineer-Databricks to drive architecture and delivery of high-throughput data pipelines on Databricks, leveraging Spark, Delta Lake, and Unity Catalog.

You will own cluster strategy, implement CI/CD, and champion secure, scalable data solutions across business functions. The role requires heavy production delivery in data engineering, with strong Python/Java skills and experience in AI-assisted engineering practices for code quality and

Qualifications

• 5+ years of software/data engineering experience with large-scale data systems.

Responsibilities

• Lead architecture and delivery of high-throughput data pipelines on Databricks with Spark.
• Establish and evolve Lakehouse patterns with Delta Lake for scalable data platforms.
• Own Databricks cluster strategy, autoscaling, and configuration.
• Automate pipelines with Databricks Workflows and Airflow, integrating with AWS services.
• Design secure ingestion/transform frameworks and governance using Unity Catalog and AWS Glue Catalog.
• Drive Spark performance tuning to optimize cost and throughput.
• Develop reusable libraries and APIs in Python/Java for data processing.
• Implement CI/CD for data projects using Git, Terraform, and automated releases.
• Promote AI-assisted engineering practices with code reviews, testing, and validation.

Skills

Databricks

Apache Spark

Python

Java

SQL

Data pipelines

CI/CD

Delta Lake

AWS

Terraform

Airflow

Security

Education

Software engineering degree

Tools

Unity Catalog

Databricks Workflows

Job description

JPMorgan Chase & Co. is seeking a Lead Software Engineer-Databricks to drive architecture and delivery of high-throughput data pipelines on Databricks, leveraging Spark, Delta Lake, and Unity Catalog.

You will own cluster strategy, implement CI/CD, and champion secure, scalable data solutions across business functions. The role requires heavy production delivery in data engineering, with strong Python/Java skills and experience in AI-assisted engineering practices for code quality and