Lead Data Platform Engineer: Databricks, Spark & Terraform

JPMorgan Chase & Co.; Chicago (IL); On-site; USD 170,000 - 250,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Lead Software Engineer to lead the Databricks-based data platform, shaping the technical roadmap and ensuring secure, scalable data products. You will mentor engineers, drive design reviews, and advance AI-assisted development practices across teams.

Responsibilities include platform architecture, data modeling, reliability, security governance, and cost optimization, with a strong emphasis on responsible AI usage and operational excellence.

Qualifications

• 7+ years in data engineering/platform engineering with hands-on Databricks in production.
• Strong proficiency in Spark (PySpark/Scala) and SQL, including performance tuning.
• Experience building and operating data platforms: batch/stream ingestion, transformations, orchestration.
• Familiarity with Data Lake design (Delta Lake, Snowflake, AWS) and optimization.
• Unity Catalog or equivalent governance tooling, permissions, catalogs/schemas, lineage.
• Solid software engineering fundamentals: Git, CI/CD, automated testing, code reviews.
• Experience implementing observability and data quality checks for pipelines and SQL workloads.
• Ability to lead technical decisions and collaborate across teams.
• Strong focus on responsible AI use in engineering workflows and secure coding practices.

Responsibilities

• Platform leadership & architecture: define the roadmap for the Databricks lakehouse platform.
• Data modeling & database engineering: design curated datasets and enforce naming/partitioning standards.
• Reliability & operations: establish SLOs, runbooks, alerting, incident response for pipelines and SQL workloads.
• Security, governance & access controls: implement Unity Catalog, least-privilege access, auditing.
• Performance & cost management: tune Spark/SQL workloads, optimize clusters, enable cost observability.
• Engineering excellence: set standards for code quality, testing, CI/CD, documentation.
• Mentorship & collaboration: coach engineers and lead design reviews with stakeholders.
• Drive adoption of AI-assisted engineering practices with secure validation standards.

Skills

Databricks

Spark PySpark/Scala

SQL

Data Platforms

Data Lake

Unity Catalog

Git

CI/CD

Automated Testing

Code Reviews

Observability

Leadership

AI-assisted development

Responsible AI

Security & Governance

Communication

Incident Response

Terraform

dbt

Delta Live Tables

Structured Streaming

Databricks SQL Warehouses

Tools

Databricks

Job description

JPMorgan Chase & Co. is seeking a Lead Software Engineer to lead the Databricks-based data platform, shaping the technical roadmap and ensuring secure, scalable data products. You will mentor engineers, drive design reviews, and advance AI-assisted development practices across teams.

Responsibilities include platform architecture, data modeling, reliability, security governance, and cost optimization, with a strong emphasis on responsible AI usage and operational excellence.