Lead Databricks Data Engineer & Spark Architect

JPMorganChase; Plano (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

JPMorganChase in Plano, TX seeks a Lead Software Engineer-Databricks to drive high-throughput data pipelines and secure, scalable data platforms. You will own Databricks clusters, implement Lakehouse patterns, and orchestrate pipelines with Airflow while ensuring data quality and governance.

You will lead performance tuning, build reusable data tooling in Python/Java, and advance AI-assisted engineering practices with a security-first mindset in a fast-paced environment.

Qualifications

• 5+ years of software/data engineering experience.
• Advanced experience with Spark on Databricks and AWS EMR.
• Expertise in Delta Lake, Unity Catalog, Workflows and SQL warehouses.
• Proven ability to design and operate reliable ETL/ELT pipelines (batch/streaming).
• Strong Python/Java programming and data modeling skills.
• Experience with AI-assisted development tools and CI/CD practices.
• Security-first mindset with data sensitivity and governance knowledge.

Responsibilities

• Lead architecture and delivery of high-throughput data pipelines on Databricks.
• Evolve Lakehouse patterns with Delta Lake for scalable data platforms.
• Manage Databricks cluster strategy, autoscaling, and configurations.
• Design secure ingestion and transformation frameworks with Airflow DAGs.
• Enforce data quality, lineage, and governance using Unity Catalog or AWS Glue.
• Tune Spark/Databricks performance for cost and throughput.
• Build reusable libraries and APIs in Python/Java with strong test coverage.
• Implement CI/CD for data projects and promote engineering standards.
• Champion AI-assisted engineering practices and secure coding standards.

Skills

Databricks

Apache Spark

Python

Java

Delta Lake

Unity Catalog

Airflow

CI/CD

AI-assisted development

Security

Tools

Terraform

Airflow

React

Job description

JPMorganChase in Plano, TX seeks a Lead Software Engineer-Databricks to drive high-throughput data pipelines and secure, scalable data platforms. You will own Databricks clusters, implement Lakehouse patterns, and orchestrate pipelines with Airflow while ensuring data quality and governance.

You will lead performance tuning, build reusable data tooling in Python/Java, and advance AI-assisted engineering practices with a security-first mindset in a fast-paced environment.