Lead Data Engineer — AI-Driven Data Platform

JPMorgan Chase & Co.; Plano (TX); On-site; USD 140,000 - 200,000; Full time

Job summary

JPMorgan Chase & Co. in Plano, TX seeks a Lead Software Engineer to design, build, and operate scalable data processing solutions using Databricks, Python, and AWS.

You will own data pipelines and a control plane for enterprise workflows, drive observability, security-by-design, and AI-assisted development while mentoring engineers across the SDLC.

Collaborate with cross-functional teams to deliver reliable data services that support business objectives and regulatory requirements.

Qualifications

• Formal training or certification in software engineering concepts with 5+ years of applied experience.
• Experience in data management and large-scale ETL/ELT processing with strong SQL, Python, and PySpark performance tuning.
• Hands-on Databricks/Spark experience and cloud data lake patterns, AWS integration.
• Proven experience building platform services/control planes including API design and config-driven systems.
• Strong focus on data quality, security-by-design, and lineage/auditability (IAM, secrets mgmt).
• Production engineering mindset: observability, monitoring, and incident response for always-on services.
• Proficient in CI/CD and release engineering with firm-standard tooling.
• Experience leading the adoption of AI-assisted development tools with governance.

Responsibilities

• Execute creative, data-driven software solutions end-to-end, solving complex technical problems.
• Design and build a control plane for enterprise data pipelines, standardizing orchestration and governance.
• Develop self-service APIs/SDKs, templates, and configuration-driven onboarding with guardrails.
• Design, develop, and maintain scalable data pipelines and processing workflows using Python, PySpark, SQL, Databricks on AWS.
• Ensure data quality, security, lineage, and observability with dashboards and automated remediation patterns.
• Lead and participate in the full SDLC, providing production support for pipeline and platform services.
• Collaborate with stakeholders to shape data management strategy and translate requirements into scalable solutions.
• Mentor engineers and foster adoption of modern engineering practices, including AI-assisted tools.
• Drive enterprise-wide AI-assisted engineering practices with validation standards and reuse of patterns.
• Apply SDLC toolchain knowledge to improve automation and value.

Skills

Data management

ETL/ELT

Python

PySpark

AI-assisted development

Security-by-design

Observability

Tools

Databricks

Spark

AWS

Terraform/CloudFormation

Jenkins

Spinnaker

Sonar

Airflow

Job description

JPMorgan Chase & Co. in Plano, TX seeks a Lead Software Engineer to design, build, and operate scalable data processing solutions using Databricks, Python, and AWS.

You will own data pipelines and a control plane for enterprise workflows, drive observability, security-by-design, and AI-assisted development while mentoring engineers across the SDLC.

Collaborate with cross-functional teams to deliver reliable data services that support business objectives and regulatory requirements.