Lead Data & ML Platform Engineer (Databricks, AWS)

慨正橡扯; Plano (TX); On-site; USD 160,000 - 210,000; Full time

Job summary

JPMorgan Chase in Plano, TX seeks a Lead Software Engineer, Machine Learning and Cloud to drive data pipelines and analytics across multiple teams. You will lead architecture, set up Databricks clusters, and advance AI-assisted development practices in a secure, scalable cloud environment.

Responsibilities include building high-throughput pipelines, Delta Lake governance, and CI/CD for data projects, with a focus on performance optimization and reliability.

Qualifications

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• 8+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.
• Demonstrated experience leading effective use of approved AI-assisted software development tools with the ability to set team expectations for validating AI outputs.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations and secure handling.
• Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.
• Hands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, SQL Warehouses).

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Spark.
• Establish lakehouse patterns with Delta Lake and ensure performance at scale.
• Drive adoption of enterprise AI-assisted engineering practices to improve quality, delivery speed, and operations.
• Own Databricks cluster strategy, including autoscaling and configuration.
• Design secure data ingestion and transformation frameworks with Databricks services.
• Enforce data quality, lineage, and governance; embed validations into pipelines.
• Build reusable libraries, frameworks, and APIs in Python/Java; oversee testing.
• Implement CI/CD for data projects and infrastructure deployments.

Skills

Python

Java

Databricks

Apache Spark

Airflow

SQL

Security

CI/CD

Leadership

AI-assisted tooling

Education

Formal software engineering training

Tools

Databricks

EMR

Delta Lake

Unity Catalog

Airflow

Terraform

Git-based workflows

Repos/notebooks

Job description

JPMorgan Chase in Plano, TX seeks a Lead Software Engineer, Machine Learning and Cloud to drive data pipelines and analytics across multiple teams. You will lead architecture, set up Databricks clusters, and advance AI-assisted development practices in a secure, scalable cloud environment.

Responsibilities include building high-throughput pipelines, Delta Lake governance, and CI/CD for data projects, with a focus on performance optimization and reliability.