Lead Data Engineer - Python, Spark & Cloud Data Pipelines

Fairygodboss; Columbus (OH); On-site; USD 140,000 - 190,000; Full time

Job summary

JPMorgan Chase is seeking a Lead Software Engineer - Data Engineering to help design and deliver trusted data products at scale within Consumer & Community Banking/Data Products. You will build large-scale ETL pipelines using Python, PySpark, Snowflake, and modern lake formats, while leveraging AWS services and AI-assisted development practices.

The role requires strong SQL, experience with DBT, and expertise in data processing frameworks, with opportunities to mentor engineers in a fast-paced

Qualifications

• Experience with Python (primary) and Java (secondary) in a large corporate environment.
• Hands-on PySpark or distributed data processing experience.
• Strong DBT and modern ETL development practices.
• Experience with workflow orchestration like Control-M or Airflow MWAA.
• Expertise with AWS data services (Glue, Athena, CloudWatch, Lambda).
• Knowledge of Iceberg or Delta Lake open table formats.
• Snowflake administration and development experience.
• Strong SQL and database skills.
• Promotes AI-assisted engineering practices and secure, high-quality code.
• Familiarity with enterprise-SDLC tooling and automation.

Responsibilities

• Design, develop, and optimize large-scale ETL data pipelines.
• Build Python applications with modularity, logging, and tests.
• Develop and maintain distributed data processing with PySpark.
• Design end-to-end tests and validation at scale.
• Implement workflow orchestration with Control-M or Apache Airflow MWAA.
• Develop cloud-native solutions using AWS services (Glue, Athena, Lambda, CloudWatch).
• Design data lake architectures with Iceberg/Delta Lake.
• Administer Snowflake environments (streams, tasks, roles, warehouses).
• Participate in code reviews, CI/CD, and engineering best practices.
• Leverage AI-assisted development tools while ensuring secure, compliant delivery.

Skills

Python

Java

PySpark

SQL

Data engineering

CI/CD

AI-assisted development

Engineering best practices

Education

Formal training or certification in software engineering

Tools

DBT

Control-M

MWAA

Snowflake

Iceberg

Delta Lake

AWS (Glue, Athena, Lambda, CloudWatch)

Airflow

Kafka

Flink

GitHub Copilot

Job description

JPMorgan Chase is seeking a Lead Software Engineer - Data Engineering to help design and deliver trusted data products at scale within Consumer & Community Banking/Data Products. You will build large-scale ETL pipelines using Python, PySpark, Snowflake, and modern lake formats, while leveraging AWS services and AI-assisted development practices.

The role requires strong SQL, experience with DBT, and expertise in data processing frameworks, with opportunities to mentor engineers in a fast-paced