Databricks Lead Engineer — Spark, Lakehouse, Pipelines

Next Frontier Capital; Wilmington (DE); On-site; USD 140,000 - 210,000; Full time

Job summary

JPMorganChase in the Corporate Sector’s Enterprise Technology team seeks a Lead Software Engineer‑Databricks to shape scalable data platforms and deliver trusted market-leading technology products securely.

You will own architecture and delivery of high‑throughput data pipelines on Databricks, optimize Spark workloads, implement CI/CD, and champion AI‑assisted engineering practices with a focus on reliability, security, and cross‑functional collaboration.

Qualifications

• 5+ years of software/data engineering with Spark on Databricks.
• Advanced Databricks expertise across Delta Lake and Unity Catalog.
• Experience building reliable batch and streaming ETL/ELT pipelines.

Responsibilities

• Lead architecture and delivery of high-throughput data pipelines on Databricks.
• Design secure ingestion and transformation frameworks with Delta Lake.
• Implement CI/CD for data projects with Git workflows and Terraform.
• Drive performance tuning and cost optimization for Spark workloads.

Skills

Apache Spark

Databricks

Python

Java

SQL

Airflow

CI/CD

Security best practices

AWS

Education

Bachelor's degree in Computer Science or related field

Tools

Delta Lake

Unity Catalog

Databricks Workflows

Airflow

Terraform

Job description

JPMorganChase in the Corporate Sector’s Enterprise Technology team seeks a Lead Software Engineer‑Databricks to shape scalable data platforms and deliver trusted market-leading technology products securely.

You will own architecture and delivery of high‑throughput data pipelines on Databricks, optimize Spark workloads, implement CI/CD, and champion AI‑assisted engineering practices with a focus on reliability, security, and cross‑functional collaboration.