Lead Data Engineer - Python, Databricks & AWS

JPMorgan Chase; Jersey City (NJ); On-site; USD 152,000 - 215,000; Full time

Job summary

JPMorgan Chase in Jersey City, NJ, is seeking a Lead Software Engineer to architect and deliver scalable data platforms on Databricks Lakehouse hosted in AWS. You will drive ingestion pipelines, implement robust testing, and mentor engineers to uphold best practices in secure, cost-aware data engineering.

You will influence design reviews, establish reliability practices, and promote responsible AI use across the team while partnering with security and platform teams to align with enterprise

Qualifications

• Formal training or certification on software engineering concepts and 5+ years hands on Software Development Life Cycle experience
• Strong data engineering experience, including proven leading delivery/architecture for multi-team data platforms.
• Hands-on experience building and operating a Databricks Lakehouse Hosted in AWS
• Deep experience with Delta Lake (ACID tables, partitioning, schema evolution
• Proven experience with Spark on Databricks (performance tuning, cluster sizing, skew mitigation, joins, caching, file sizing).
• Experience with streaming and batch pipelines (Structured Streaming; incremental processing; backfills; late-arriving data).
• Strong AWS fundamentals for data platforms: S3, IAM, KMS, networking basics (VPC/security groups), logging/auditing.
• Experience implementing data governance/security controls in Databricks (e.g., Unity Catalog, table/column permissions, credential passthrough patterns as applicable).
• Demonstrated experience leading effective use of approved AI-assisted software development tools with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations
• Demonstrated ownership of reliability: monitoring/alerting, incident response, RCA, and SLO/SLA management.

Responsibilities

• Architect the lake house: design bronze/silver/gold layers and data products
• Deliver ingestion at scale from AWS into Databricks (batch + streaming; include CDC)
• Build maintainable pipelines with Delta Live Tables or Jobs; ensure testing and docs
• Operational excellence: productionize workloads with workflows, retries, checkpointing, idempotency
• Governance by design: enforce least privilege, PII classification, auditing, lineage
• Promote enterprise AI-assisted engineering practices to improve code quality and delivery speed
• Apply SDLC toolchain expertise, including AI-assisted development and automation
• Performance & cost management: tune Spark/Delta workloads, optimize storage, cluster sizing
• Lead and mentor: set standards, review designs, upskill engineers, ensure security alignment
• CI/CD and IaC: Terraform for Databricks + AWS; promote across environments
• Testing: unit/integration tests, data quality checks, replay/backfill procedures

Skills

Databricks Lakehouse

Spark / Delta Lake optimization

AWS fundamentals

Terraform / IaC

Leadership / mentoring

CI/CD automation

Data governance / security

AI-assisted development practices

Education

Formal software engineering training or certification

Experience with SDLC

Tools

Databricks

Delta Lake

Terraform

AWS (S3/IAM/KMS)

Unity Catalog / data governance

Job description

JPMorgan Chase in Jersey City, NJ, is seeking a Lead Software Engineer to architect and deliver scalable data platforms on Databricks Lakehouse hosted in AWS. You will drive ingestion pipelines, implement robust testing, and mentor engineers to uphold best practices in secure, cost-aware data engineering.

You will influence design reviews, establish reliability practices, and promote responsible AI use across the team while partnering with security and platform teams to align with enterprise