Lead Databricks Data Platform Architect

JPMorgan Chase & Co.; Plano (TX); On-site; USD 150,000 - 190,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Lead Software Engineer-Databricks to join our Corporate Technology team in Plano, TX.

You will be a core technical contributor responsible for building high-throughput data pipelines and trusted data platforms using Databricks, Spark, Delta Lake, and related tooling. Responsibilities include designing scalable data architectures, tuning Spark workloads, and implementing CI/CD with Git and Terraform.

Qualifications

• 5+ years of software/data engineering experience.
• ProductionDelivery with Apache Spark on Databricks and/or AWS EMR.
• Expertise in Delta Lake, Unity Catalog, and SQL Warehouses.
• Proven ability to design and operate reliable ETL/ELT data pipelines.
• Strong programming in Python and/or Java.

Responsibilities

• Lead the architecture and delivery of high-throughput, low-latency data pipelines on Databricks using Apache Spark.
• Establish and evolve Lakehouse patterns with Delta Lake for scalable data platforms.
• Own Databricks cluster strategy and configuration including autoscaling and sizing.
• Orchestrate pipelines using Databricks Workflows and integrate with AWS services.
• Design secure ingestion and transformation frameworks leveraging Databricks services and Airflow DAGs.
• Enforce data quality, lineage, and governance using Unity Catalog or AWS Glue Catalog.
• Drive Spark and Databricks performance tuning to optimize cost and throughput.
• Build and maintain reusable libraries and APIs in Python and/or Java.
• Implement CI/CD for data projects using Git, Terraform, and automated releases.

Skills

Spark

Python

Java

Data engineering

Tools

Databricks

Delta Lake

Unity Catalog

Workflows

Airflow

Terraform

AWS

SQL

Job description

JPMorgan Chase & Co. is seeking a Lead Software Engineer-Databricks to join our Corporate Technology team in Plano, TX.

You will be a core technical contributor responsible for building high-throughput data pipelines and trusted data platforms using Databricks, Spark, Delta Lake, and related tooling. Responsibilities include designing scalable data architectures, tuning Spark workloads, and implementing CI/CD with Git and Terraform.