Lead Spark & Data Engineer — PySpark & Lakehouse

Next Frontier Capital; Jersey City (NJ); On-site; USD 150,000 - 190,000; Full time

Job summary

JPMorganChase, within the CIB Regulatory Reporting Team, seeks a Lead Software Engineer to drive production data engineering and Spark-based solutions in a secure, scalable lakehouse environment. You will work as a core technical contributor across business functions to support the firm’s objectives.

You will collaborate with agile teams to design, implement, and optimize PySpark pipelines on EMR/Databricks, apply AI-assisted tooling, and guide engineers while upholding governance, security, and

Qualifications

• 5+ years of applied experience building production data engineering or software solutions.
• Experience delivering Spark workloads on EMR and/or Databricks.
• Production data pipelines with data modeling and testing.
• Strong Python and PySpark with SQL fundamentals.
• Experience using AI-assisted development tools with governance.
• AWS data management patterns including S3 and Glue Data Catalog.
• Iceberg lakehouse production design, partitioning, and schema evolution.
• Automation and CI/CD for data pipelines.
• Leadership in agile teams and best practices.

Responsibilities

• Executes creative software solutions with focus on data engineering and Spark-based ETL/ELT.
• Develops production Python/PySpark code and reviews others' Spark jobs and SQL logic.
• Drives AI-assisted engineering practices to improve quality and delivery speed with validation standards.
• Applies SDLC tools and automation to improve value and reuse patterns.
• Eliminates recurring issues to improve stability of data pipelines and lakehouse maintenance.
• Leads evaluation sessions with vendors and teams on architecture and applicability.
• Leads communities of practice around Spark, Iceberg, and data platform operations.
• Contributes to team culture of diversity, inclusion, and respect.

Skills

Python

PySpark

Spark SQL

SQL fundamentals

AI-assisted development tools

AWS data platforms

Databricks / EMR

Iceberg lakehouse

CI/CD for data pipelines

Leadership

Tools

Databricks

EMR

Iceberg

S3

Glue Data Catalog

CI/CD tooling

Job description

JPMorganChase, within the CIB Regulatory Reporting Team, seeks a Lead Software Engineer to drive production data engineering and Spark-based solutions in a secure, scalable lakehouse environment. You will work as a core technical contributor across business functions to support the firm’s objectives.

You will collaborate with agile teams to design, implement, and optimize PySpark pipelines on EMR/Databricks, apply AI-assisted tooling, and guide engineers while upholding governance, security, and