Lead Software Engineer - Python, Databricks and AWS

JPMorgan Chase & Co.; Jersey City (NJ); On-site; USD 180,000 - 210,000; Full time

Job summary

JPMorgan Chase & Co. in New Jersey is seeking a Lead Software Engineer to drive scalable data platform solutions in Corporate Technology.

You will architect and implement a lakehouse, manage large-scale data ingestion from AWS into Databricks, and champion AI-assisted engineering practices across multi-team delivery. You will mentor engineers, lead design reviews, and ensure secure, compliant data handling while optimizing performance and cost.

Qualifications

• 5+ years of hands-on SDLC experience in software engineering.
• Extensive data engineering leadership for multi-team data platforms.
• Hands-on building and operating a Databricks Lakehouse on AWS.
• Deep Delta Lake expertise including ACID tables and schema evolution.
• Experience with streaming and batch pipelines (Structured Streaming).
• Strong AWS fundamentals: S3, IAM, KMS.
• Experience with AI-assisted development tools and responsible AI practices.
• Reliability ownership: monitoring, incident response, RCA, SLO/SLA.

Responsibilities

• Architect the lake house: design bronze/silver/gold layers and domain data products.
• Deliver ingestion at scale from AWS sources into Databricks (batch + streaming).
• Build maintainable pipelines with Delta Live Tables or modular jobs.
• Operational excellence: productionize workloads with retries, checkpointing, idempotency.
• Governance by design: enforce least privilege, data classification, auditing, lineage.
• Lead adoption of enterprise AI-assisted engineering practices and ensure code quality.
• Apply SDLC tools to improve automation value.
• Performance & cost management: tune Spark/Delta workloads and clusters.
• Lead and mentor: set standards and upskill engineers in Spark/Databricks.
• CI/CD and IaC: Terraform for Databricks + AWS resources; promote across environments.
• Testing: unit/integration tests, data quality checks, version control, runbooks.

Skills

Data engineering

Lakehouse architecture

Databricks

Spark tuning

AWS data services

CI/CD

IaC Terraform

Security & governance

Mentoring

Data governance

Tools

Databricks

Delta Lake

Terraform

AWS (S3/IAM)

CI/CD tooling

Job description

We have an opportunity to impact your career and provide an adventure where you can push the limits of what's possible.

As a Lead Software Engineer at JPMorgan Chase within the Corporate Technology, you are an integral part of an agile team that works to enhance, build, and deliver trusted market-leading technology products in a secure, stable, and scalable way. As a core technical contributor, you are responsible for conducting critical technology solutions across multiple technical areas within various business functions in support of the firm’s business objectives.

Job responsibilities

• Architect the lake house: design bronze/silver/gold (or equivalent) layers, domain data products
• Deliver ingestion at scale: implement resilient ingestion from AWS sources into Databricks (batch + streaming), including CDC where needed.
• Build maintainable pipelines: use Delta Live Tables (DLT) and/or standard Jobs with clear modular structure, testing, and documentation.
• Operational excellence: productionize workloads via Databricks Workflows/Jobs, robust retries, checkpointing, idempotency, and safe re-runs.
• Governance by design: enforce least privilege, data classification (PII), auditing, lineage/metadata, and controlled sharing/consumption.
• Drives team adoption of enterprise-authorized AI-assisted engineering practices within the work environment to improve code quality, delivery speed, and operational outcomes (e.g., AI-assisted code review/refactoring, test strategy acceleration, incident/root-cause analysis support), while establishing consistent validation standards (secure coding, peer review, automated testing) and promoting reuse of effective patterns across the team.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation.
• Performance & cost management: tune Spark/Delta workloads, right-size clusters, optimize storage layout, and manage job/warehouse spend.
• Lead and mentor: set engineering standards, run design reviews, drive code quality, and upskill engineers in Spark/Databricks best practices, cross-functional delivery: translate stakeholder needs into technical plans, communicate tradeoffs, and align with security/platform teams.
• CI/CD and IaC: Terraform (preferred) for Databricks + AWS resources; promotion across environments.
• Testing: unit/integration tests for transformations, data quality checks, contract testing, and replay/backfill procedures, version control & code review discipline; clear documentation and runbooks.

Required qualifications, capabilities, and skills

• Formal training or certification on software engineering concepts and 5+ years hands on Software Development Life Cycle experience
• Strong data engineering experience, including proven leading delivery/architecture for multi-team data platforms.
• Hands-on experience building and operating a Databricks Lakehouse Hosted in AWS
• Deep experience with Delta Lake (ACID tables, partitioning, schema evolution,
• Proven experience with Spark on Databricks (performance tuning, cluster sizing, skew mitigation, joins, caching, file sizing).
• Experience with streaming and batch pipelines (Structured Streaming; incremental processing; backfills; late-arriving data).
• Strong AWS fundamentals for data platforms: S3, IAM, KMS, networking basics (VPC/security groups), logging/auditing.
• Experience implementing data governance/security controls in Databricks (e.g., Unity Catalog, table/column permissions, credential passthrough patterns as applicable).
• Demonstrated experience leading effective use of approved AI-assisted software development tools (e.g., for coding, code review, test acceleration, troubleshooting) with the ability to set team expectations for validating AI outputs for correctness, performance, and security.
• Strong understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; experience coaching engineers on safe, compliant adoption within delivery practices
• Demonstrated ownership of reliability: monitoring/alerting, incident response, RCA, and SLO/SLA management.