Senior Engineering Lead: Databricks, AI‑Driven Data Pipelines

JPMorganChase; Plano (TX); On-site; USD 180,000 - 240,000; Full time

Job summary

JPMorganChase in Plano, TX, seeks a Manager of Software Engineering to guide multiple teams in delivering trusted, scalable technology within Corporate Sector, Enterprise Technology. You will lead architecture, data pipelines, and AI-assisted engineering practices to improve quality and delivery speed.

The role demands deep Databricks/Spark expertise, strong Python/Java, and a track record building ETL/ELT pipelines with secure, reliable operations. Excellent communication required.

Qualifications

• 5+ years of software engineering experience.
• 10+ years of professional software/data engineering experience with Spark on Databricks or EMR.
• Experience using AI-assisted development tools with validation of outputs for correctness, performance, and security.
• Strong proficiency in Python and/or Java for data processing and automation.
• Databricks expertise including Delta Lake, Unity Catalog, and Workflows.
• Proven track record building ETL/ELT pipelines with SLAs and reliability engineering.
• Security-first mindset with data protection and network controls.

Responsibilities

• Lead architecture and delivery of high-throughput data pipelines using Databricks and Spark.
• Establish lakehouse patterns with Delta Lake and ensure performance at scale.
• Promote enterprise AI-assisted engineering practices to improve code quality and delivery speed.
• Apply SDLC toolchain knowledge to improve automation and value.
• Own Databricks cluster strategy, sizing, and policies.
• Orchestrate jobs with Databricks Workflows and integrate with AWS eventing.
• Design secure data ingestion and transformation frameworks using Databricks services.
• Enforce data quality, lineage, and governance via Unity Catalog/Glue Catalog.
• Advance Spark performance tuning and cluster optimization for cost efficiency.
• Build reusable libraries and APIs in Python/Java; oversee testing and validation.
• Implement CI/CD for data projects and infrastructure as code.

Skills

Python

Java

Databricks

Apache Spark

SQL

CI/CD

Airflow

Terraform

Unity Catalog

Glue Catalog

Data engineering

Security

Education

Formal training or certification in software engineering

Tools

None

Job description

JPMorganChase in Plano, TX, seeks a Manager of Software Engineering to guide multiple teams in delivering trusted, scalable technology within Corporate Sector, Enterprise Technology. You will lead architecture, data pipelines, and AI-assisted engineering practices to improve quality and delivery speed.

The role demands deep Databricks/Spark expertise, strong Python/Java, and a track record building ETL/ELT pipelines with secure, reliable operations. Excellent communication required.