Senior Lead Data Engineer - Databricks, Spark & AWS

JPMorgan Chase; Columbus (OH); On-site; USD 150,000 - 190,000; Full time

Job summary

JPMorganChase in Columbus is seeking a Lead Software Engineer to guide multiple teams in building high-performance, secure data platforms and scalable software solutions.

You will own Databricks architectures, drive Spark tuning, and champion AI-assisted development while delivering reliable ETL/ELT pipelines and collaborating with cross-functional teams across the firm.

This role emphasizes AWS data services, CI/CD, and mentoring engineers to uphold strict security and quality standards.

Qualifications

• 10+ years of professional software/data engineering experience.
• Experience with Spark on Databricks or EMR.
• Experience leading AI-assisted software development tools with proper validation of AI outputs.
• Strong Python/Java for data processing and automation.
• Robust SQL and data modeling skills (dimensional/star schema).
• AWS experience: S3, IAM, Glue, CloudWatch, Kinesis/MSK, DynamoDB.
• Proven track record of designing and operating ETL/ELT pipelines.
• CI/CD and testing frameworks (pytest, JUnit).
• Security-first mindset: secure handling of data, credentials, encryption, access controls.

Responsibilities

• Lead architecture and delivery of data pipelines with Databricks and Spark.
• Establish lakehouse patterns with Delta Lake and ensure performance at scale.
• Promote AI-assisted engineering practices with secure coding and automated testing.
• Own Databricks cluster strategy, autoscaling, configs, and policies.
• Orchestrate Databricks Workflows and integrate with AWS services.
• Design secure data ingestion and transformation frameworks using AWS.
• Drive Spark performance tuning and cost optimization; build reusable libraries and APIs.
• Implement CI/CD for data projects and Terraform deployments.

Skills

Python

Java

Databricks

Spark

AI-assisted development

Security

SQL

AWS

ETL/ELT pipelines

CI/CD

Education

Formal training or certification in software engineering

Tools

Terraform

Job description

JPMorganChase in Columbus is seeking a Lead Software Engineer to guide multiple teams in building high-performance, secure data platforms and scalable software solutions.

You will own Databricks architectures, drive Spark tuning, and champion AI-assisted development while delivering reliable ETL/ELT pipelines and collaborating with cross-functional teams across the firm.

This role emphasizes AWS data services, CI/CD, and mentoring engineers to uphold strict security and quality standards.