Lead Data Platform Engineer - Databricks on AWS

Next Frontier Capital; Kentucky; On-site; USD 180,000 - 230,000; Full time

Job summary

JPMorgan Chase & Co. is seeking a Lead Software Engineer to guide multiple teams in delivering scalable data products within Corporate Sector. You will own Databricks architectures, build high-throughput pipelines, and shape governance, security, and reliability across cloud data platforms.

The role requires deep expertise in Spark, Delta Lake, AWS, and CI/CD, plus leadership to steer design reviews and cross-functional collaboration. Competitive rewards accompany this senior position.

Qualifications

• 5+ years applied software/data engineering experience.
• 10+ years professional software/data engineering experience, including Spark on Databricks or EMR.
• Strong proficiency in Python and/or Java for data processing, platform tooling, and automation.
• Hands-on Databricks expertise (Delta Lake, Unity Catalog, Workflows, Repos/notebooks, SQL Warehouses).
• Solid AWS experience: S3, IAM, Glue, CloudWatch, Kinesis / MSK, DynamoDB.
• Proven track record architecting and operating ETL/ELT pipelines (batch and streaming).
• Deep skills in Spark performance tuning and Databricks cluster setup/optimization.
• Strong SQL and analytics data modeling (dimensional/star schema; lakehouse best practices).
• CI/CD and automation tooling for data (Git workflows, artifact management) and testing frameworks (pytest, JUnit).
• Security-first mindset: roles/instance profiles, secret management, encryption-at-rest/in-transit, and network controls.
• Demonstrated leadership in code quality, reviews, testing strategy, CI/CD, and technical mentorship.

Responsibilities

• Lead architecture and delivery of high-throughput, low-latency data pipelines using Databricks and Apache Spark (Core, SQL, Structured Streaming).
• Establish lakehouse patterns with Delta Lake and ensure performance at scale.
• Own Databricks cluster strategy and setup: runtime selection, autoscaling, driver/executor sizing, Spark configurations, unit scripts, cluster policies, pools, and instance profiles.
• Orchestrate jobs with Databricks Workflows; integrate with AWS eventing and orchestration as needed.
• Design secure data ingestion and transformation frameworks leveraging AWS services: S3 for data lake storage, Glue for catalog/metadata, IAM and Secrets Manager, CloudWatch, Lambda, Kinesis/MSK.
• Enforce data quality, lineage, and governance using Unity Catalog and Glue Catalog; embed validation into pipelines.
• Drive Spark performance engineering: partitioning, AQE, broadcast joins, shuffle tuning, caching, and cost-aware tuning.
• Build reusable libraries, frameworks, and APIs in Python and/or Java; oversee unit/integration/data validation testing.
• Implement CI/CD for data projects; Terraform deployments, environment promotion, and automated deployments; uphold engineering standards and code reviews.
• Partner with security and networking teams to enforce encryption, network controls, and least-privilege access; ensure policy compliance.
• Lead incident response, root-cause analysis, establish SLAs, observability, runbooks, and drive reliability/cost improvements.
• Mentor engineers, contribute to design reviews, roadmaps, backlog prioritization; collaborate with product owners and cross-functional teams.

Skills

Databricks

Python/Java

Spark

AWS

Data pipelines

SQL

Leadership

CI/CD

Security

Tools

Terraform

Git workflows

Job description

JPMorgan Chase & Co. is seeking a Lead Software Engineer to guide multiple teams in delivering scalable data products within Corporate Sector. You will own Databricks architectures, build high-throughput pipelines, and shape governance, security, and reliability across cloud data platforms.

The role requires deep expertise in Spark, Delta Lake, AWS, and CI/CD, plus leadership to steer design reviews and cross-functional collaboration. Competitive rewards accompany this senior position.