Lead Data Engineer — Databricks & AI-Driven Platform

JPMorganChase; Plano (TX); On-site; USD 150,000 - 210,000; Full time

Job summary

JPMorganChase in Plano, TX seeks a Lead Software Engineer to architect and deliver high-throughput data pipelines on Databricks, driving performance and scalability across Spark cores, SQL, and streaming workloads.

You will own Databricks cluster strategy, enforce data governance with Unity Catalog, and promote AI-assisted development practices while ensuring secure, compliant delivery. Strong Python/SQL skills and 5+ years in data engineering are expected.

Qualifications

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• Advanced experience in software engineering and Data Engineering, including significant production delivery with Apache Spark on Databricks and/or AWS EMR.
• Demonstrated experience leading effective use of approved AI-assisted software development tools with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices.
• Advanced hands-on Databricks expertise across Delta Lake, Unity Catalog, Workflows, Repos/notebooks, and SQL Warehouses, including cluster configuration and optimization.
• Proven ability to architect, build, & operate reliable ETL/ELT data pipelines (batch and streaming), including schema design/evolution, SLAs, and reliability engineering practices.
• Deep Spark performance tuning skills, with experience diagnosing bottlenecks and optimizing jobs for scalability, cost, and runtime efficiency.
• Strong programming proficiency in Python for data processing, platform tooling, and automation.
• Strong SQL and analytics data modeling expertise, including dimensional/star schema design and Lakehouse best practices.

Responsibilities

• Lead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark (Core, SQL, Structured Streaming).
• Establish and evolve Lakehouse patterns with Delta Lake to ensure performant data platforms at scale.
• Drive AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes.
• Own Databricks cluster strategy, including runtime selection, autoscaling, sizing, and policies.
• Orchestrate and automate pipelines using Databricks Workflows and integrate with AWS services as needed.
• Design secure ingestion and transformation frameworks using Databricks, Delta, and Airflow DACs.
• Enforce data quality, lineage, and governance using Unity Catalog or AWS Glue Catalog.
• Drive Spark and Databricks performance engineering to optimize cost and throughput.
• Build reusable libraries, frameworks, and APIs in Python/Java with solid test coverage.
• Implement CI/CD for data projects using Git-based workflows and Terraform for infra deployments.

Skills

Databricks

Spark

Python

SQL

Data Engineering

CI/CD

AI-assisted engineering

Security

Education

Formal software engineering training

Tools

Delta Lake

Unity Catalog

Airflow

Terraform

Databricks Workflows

Repos/notebooks

Job description

JPMorganChase in Plano, TX seeks a Lead Software Engineer to architect and deliver high-throughput data pipelines on Databricks, driving performance and scalability across Spark cores, SQL, and streaming workloads.

You will own Databricks cluster strategy, enforce data governance with Unity Catalog, and promote AI-assisted development practices while ensuring secure, compliant delivery. Strong Python/SQL skills and 5+ years in data engineering are expected.