Lead Databricks & Spark Data Engineer

Next Frontier Capital; Plano (TX); On-site; USD 150,000 - 210,000; Full time

Job summary

JPMorgan Chase's Corporate Technology team seeks a Lead Software Engineer-Databricks to advance data pipelines with high throughput and low latency. You will be a core technical contributor shaping architecture across Spark, Delta Lake, and cloud integrations.

You will own Databricks cluster strategy, build reusable Python/Java libraries, implement CI/CD, and promote secure, auditable data processing with AI-assisted tooling.

Qualifications

• Formal training or certification on software engineering concepts.
• 5+ years of applied experience in software/data engineering.
• Advanced Spark/Databricks experience in production systems.

Responsibilities

• Lead architecture and delivery of high-throughput data pipelines on Databricks.
• Own Databricks cluster strategy, autoscaling, and configurations.
• Design secure ingestion and transformation frameworks with Delta Lake.
• Automate pipelines with Databricks Workflows and Airflow DAGs.
• Develop reusable libraries in Python/Java and ensure test coverage.

Skills

Databricks

Apache Spark

Python

Java

SQL

Airflow

CI/CD

Education

Software engineering certification

Tools

Delta Lake

Unity Catalog

Terraform

Git

Airflow

Job description

JPMorgan Chase's Corporate Technology team seeks a Lead Software Engineer-Databricks to advance data pipelines with high throughput and low latency. You will be a core technical contributor shaping architecture across Spark, Delta Lake, and cloud integrations.

You will own Databricks cluster strategy, build reusable Python/Java libraries, implement CI/CD, and promote secure, auditable data processing with AI-assisted tooling.