Lead Data Engineer: Databricks & Spark on AWS

Testing Site - JPMC; Plano (TX); On-site; USD 140,000 - 210,000; Full time

Job summary

JPMorgan Chase in Plano, TX seeks a Lead Software Engineer to guide multiple teams in building, deploying, and securing scalable data products. You will drive architecture for high-throughput pipelines using Databricks, Spark, and Delta Lake, while ensuring governance, reliability, and cost efficiency across business functions.

You will oversee cluster strategy, CI/CD, and collaboration with security and networking teams to enforce encryption and least-privilege access.

Qualifications

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• 10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.
• Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).
• Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configs, init scripts, cluster policies, pools, and instance profiles.
• Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.
• Design secure data ingestion and transformation frameworks leveraging AWS services: S3, Glue, IAM, Secrets Manager, CloudWatch, Lambda, Kinesis/MSK.
• Enforce data quality, lineage, and governance using Unity Catalog and Glue Catalog; embed expectations and validation into pipelines.
• Drive Spark performance engineering and cost optimization; build reusable libraries, APIs in Python/Java; oversee testing.
• Implement CI/CD for data projects; Terraform deployments, environment promotion, automated deployments; enforce standards and reviews.
• Partner with security and networking teams to enforce encryption and least-privilege access; ensure policy compliance.

Skills

Python

Java

Databricks

Spark

AWS

Data pipelines

SQL

Leadership

Tools

Databricks

EMR

Terraform

Job description

JPMorgan Chase in Plano, TX seeks a Lead Software Engineer to guide multiple teams in building, deploying, and securing scalable data products. You will drive architecture for high-throughput pipelines using Databricks, Spark, and Delta Lake, while ensuring governance, reliability, and cost efficiency across business functions.

You will oversee cluster strategy, CI/CD, and collaboration with security and networking teams to enforce encryption and least-privilege access.