Lead Data Engineer – Databricks & Spark on AWS

JPMorganChase; Columbus (OH); On-site; USD 140,000 - 190,000; Full time

Job summary

JPMorganChase is seeking a Lead Software Engineer in Columbus to guide multiple teams in delivering trusted market-leading technology products. You will lead architecture, data pipelines, and secure data frameworks across AWS services, Databricks, and Spark, while championing AI-assisted development practices and scalable solutions.

The role requires extensive Java/Python experience, strong data engineering credentials, and a focus on secure, high-performance delivery.

Qualifications

• Formal training or certification on software engineering concepts and 5+ years applied experience.
• 10+ years of professional software/data engineering experience, including substantial production work with Spark on Databricks or EMR.
• Experience leading effective use of approved AI-assisted software development tools with validated outputs.
• Strong proficiency in Python and/or Java for data processing and automation.
• Security-first mindset: secure handling of data, access control, and resiliency practices.

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines.
• Establish lakehouse patterns with Delta Lake and ensure scale performance.
• Promote AI-assisted engineering practices and standardized validation.
• Oversee CI/CD for data projects and infrastructure deployments.
• Own Databricks cluster strategy, Spark configurations, and job orchestration.
• Design secure data ingestion frameworks using AWS services.
• Drive Spark performance tuning and cost-optimized deployment.
• Build reusable libraries and APIs in Python/Java; oversee testing.

Skills

Python

Java

Databricks

Spark

AWS

CI/CD

Data engineering

Leadership

Tools

Databricks

Spark

Unity Catalog

Glue Catalog

Git

Terraform

AWS

Job description

JPMorganChase is seeking a Lead Software Engineer in Columbus to guide multiple teams in delivering trusted market-leading technology products. You will lead architecture, data pipelines, and secure data frameworks across AWS services, Databricks, and Spark, while championing AI-assisted development practices and scalable solutions.

The role requires extensive Java/Python experience, strong data engineering credentials, and a focus on secure, high-performance delivery.