Senior Data Pipeline Engineer: Python, Databricks & AWS

JPMorgan Chase & Co.; Jersey City (NJ); On-site; USD 140,000 - 180,000; Full time

Job summary

JPMorgan Chase & Co. in Jersey City seeks a Software Engineer to join Corporate Technology - Global Finance Technology. You will maintain data pipelines and architectures across multiple business functions, enabling secure, scalable data collection, storage, and analytics.

The role focuses on building resilient ingestion, utilizing Databricks with AWS, implementing governance controls, and driving automated testing and deployment with IaC. Strong collaboration in an agile team is expected.

Qualifications

• Formal training or certification on software engineering concepts and 3+ years hands on Software Development Life Cycle experience.
• Experience in data engineering experience building and maintaining data pipelines.
• Hands-on experience building and operating a Databricks Lakehouse Hosted in AWS.
• Experience with Delta Lake (ACID tables, partitioning, schema evolution).
• Proven experience with Spark on Databricks (performance tuning, cluster sizing, skew mitigation, joins, caching, file sizing).
• Experience with streaming and batch pipelines (Structured Streaming; incremental processing; backfills; late-arriving data).
• Strong AWS fundamentals for data platforms: S3, IAM, KMS, networking basics (VPC/security groups), logging/auditing.

Responsibilities

• Deliver ingestion at scale: implement resilient ingestion from AWS sources into Databricks (batch + streaming).
• Build maintainable pipelines using Delta Live Tables or standard Jobs with modular structure, testing, and docs.
• Operationalize workloads via Databricks Workflows/Jobs with retries, checkpointing, idempotency, and safe re-runs.
• Governance by design: enforce least privilege, data classification, auditing, lineage/metadata, controlled sharing.
• Performance & cost management: tune Spark/Delta workloads, right-size clusters, optimize storage layout, manage spend.
• CI/CD and IaC: Terraform for Databricks + AWS; promotion across environments; unit/integration tests and data quality checks.
• Version control & code review discipline; clear documentation and runbooks.
• Leverages enterprise-authorized AI coding assist tools to improve quality and speed, validated by peer review and secure coding standards.
• Applies knowledge of SDLC toolchain including AI-assisted development to improve automation.

Skills

SDLC experience

Data pipelines

Databricks Lakehouse

Delta Lake

Spark on Databricks

AWS basics

Data governance

AI-assisted tooling

Tools

Terraform

Databricks

AWS

Unity Catalog

S3

IAM

KMS

Job description

JPMorgan Chase & Co. in Jersey City seeks a Software Engineer to join Corporate Technology - Global Finance Technology. You will maintain data pipelines and architectures across multiple business functions, enabling secure, scalable data collection, storage, and analytics.

The role focuses on building resilient ingestion, utilizing Databricks with AWS, implementing governance controls, and driving automated testing and deployment with IaC. Strong collaboration in an agile team is expected.