Senior Data Engineer: Python, Databricks & AWS

JPMorganChase; Jersey City (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

JPMorganChase is seeking a Software Engineer within Corporate Technology to advance data collection, storage, and analytics solutions. You will maintain critical data pipelines across multiple business functions, leveraging Databricks on AWS and governing data with Delta Lake and Unity Catalog.

You will work on scalable ingestion, CI/CD, and AI-assisted development while ensuring security, reliability, and cost efficiency across production workloads.

Qualifications

• Formal training or certification on software engineering concepts and 3+ years hands on Software Development Life Cycle experience.
• Experience in data engineering experience building and maintaining data pipelines.
• Hands-on experience building and operating a Databricks Lakehouse hosted in AWS.
• Experience with Delta Lake (ACID tables, partitioning, schema evolution).
• Proven experience with Spark on Databricks (performance tuning, cluster sizing, skew mitigation, joins, caching, file sizing).
• Experience with streaming and batch pipelines (Structured Streaming; incremental processing; backfills; late-arriving data).
• Strong AWS fundamentals for data platforms: S3, IAM, KMS, networking basics (VPC/security groups), logging/auditing.
• Experience implementing data governance/security controls in Databricks (e.g., Unity Catalog, table/column permissions, credential passthrough patterns as applicable)
• Demonstrate experience with reliability: monitoring/alerting, incident response, RCA, and SLO/SLA management
• Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices.
• Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security)

Responsibilities

• Deliver ingestion at scale: implement resilient ingestion from AWS sources into Databricks (batch + streaming).
• Build maintainable pipelines: use Delta Live Tables (DLT) and/or standard Jobs with clear modular structure, testing, and documentation.
• Operational excellence: productionize workloads via Databricks Workflows/Jobs, robust retries, checkpointing, idempotency, and safe re-runs.
• Governance by design: enforce least privilege, data classification (PII), auditing, lineage/metadata, and controlled sharing/consumption.
• Performance & cost management: tune Spark/Delta workloads, right-size clusters, optimize storage layout, and manage job/warehouse spend.
• CI/CD and IaC: Terraform (preferred) for Databricks + AWS resources; promotion across environments. Testing: unit/integration tests for transformations, data quality checks, contract testing, and replay/backfill procedures.
• Version control & code review discipline; clear documentation and runbooks
• Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity; validate AI outputs via peer review and testing.
• Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities to improve automation value.

Skills

SDLC experience

Data pipelines

Databricks

Delta Lake

Spark tuning

AWS fundamentals

Tools

Databricks

Delta Live Tables

Terraform

AWS

Job description

JPMorganChase is seeking a Software Engineer within Corporate Technology to advance data collection, storage, and analytics solutions. You will maintain critical data pipelines across multiple business functions, leveraging Databricks on AWS and governing data with Delta Lake and Unity Catalog.

You will work on scalable ingestion, CI/CD, and AI-assisted development while ensuring security, reliability, and cost efficiency across production workloads.