Lead Data Engineering Architect, AI-Driven Pipelines

慨正橡扯; Plano (TX); On-site; USD 140,000 - 200,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Lead Software Engineer to design, develop, and maintain scalable data processing solutions on Databricks with Python and AWS. You will lead data pipelines, define a control plane, and drive governance, security, and observability across analytics workloads.

You will mentor engineers, collaborate with cross-functional teams, and promote responsible AI-assisted development practices to accelerate delivery and ensure reliability.

Qualifications

• 5+ years in software engineering with data processing focus.
• Strong SQL, Python and PySpark for large-scale ETL/ELT workloads.
• Experience with Databricks/Spark on AWS and cloud data lakes.
• Design of platform services, APIs, and configuration-driven systems.
• Knowledge of data quality, security-by-design and lineage.
• Proven ability to lead SDLC, observability, and incident response.

Responsibilities

• Develop end-to-end, data-driven software solutions and troubleshoot complex issues.
• Design a control plane for enterprise data pipelines with governance and run-time management.
• Build self-service APIs/SDKs, templates, and onboarding with guardrails.
• Create scalable data pipelines and models for analytics on AWS Databricks.
• Ensure data quality, security, lineage, and operational transparency.
• Lead SDLC activities and provide production support to improve reliability.
• Mentor engineers and promote AI-assisted development practices.
• Adopt responsible AI practices and coach teams on safe AI adoption.

Skills

Databricks

Python

AWS

SQL

PySpark

Data modeling

CI/CD

Observability

AI-assisted tooling

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks/Spark

Terraform/CloudFormation

Airflow

Jenkins

Spinnaker

Sonar

Job description

JPMorgan Chase & Co. is seeking a Lead Software Engineer to design, develop, and maintain scalable data processing solutions on Databricks with Python and AWS. You will lead data pipelines, define a control plane, and drive governance, security, and observability across analytics workloads.

You will mentor engineers, collaborate with cross-functional teams, and promote responsible AI-assisted development practices to accelerate delivery and ensure reliability.