Lead Software Engineer - Databricks, Spark

JPMorgan Chase & Co.; Chicago (IL); In loco; USD 61.000 - 101.000; Tempo pieno

Descrizione del lavoro

JPMorgan Chase & Co. is seeking a Lead Software Engineer to guide a Databricks lakehouse platform, spanning ingestion, storage, modeling, and serving with clear standards.

You will design curated datasets, tune Spark/SQL workloads, implement governance, and mentor engineers while advancing responsible AI-assisted development and secure practices.

Join a regulated-data environment in Chicago for a high-impact role driving enterprise-grade data solutions.

Competenze

• 7+ years of experience in data or platform engineering, including Databricks production work.
• Strong Spark (PySpark/Scala) and SQL with performance tuning skills.
• Built and operated data platforms with batch/streaming ingestion and orchestration.
• In-depth knowledge of data lake table design across Databricks, Snowflake, or AWS.

Mansioni

• We define and steer the technical roadmap for the Databricks lakehouse platform.
• We design curated datasets and enforce naming, partitioning, and performance standards.
• We tune Spark/SQL workloads and manage clusters/warehouses with cost observability.
• We implement governance, access control, auditing, and data lifecycle management.
• We establish code quality, CI/CD standards, and reusable libraries.
• We coach engineers and translate business needs into scalable data capabilities.

Conoscenze

Databricks

Spark

SQL

Python

Scala

Git

CI/CD

Terraform

dbt

Snowflake

AWS

AI-assisted development

Telemetry/Observability

Security governance

Strumenti

Databricks

Snowflake

Terraform

dbt

GitHub

Copilot

Descrizione del lavoro

Salary: $61,000 - 101,000 per year

Requirements:

• We bring at least 7 years of experience in data engineering or platform engineering, including hands-on Databricks work in production settings.
• We have strong proficiency in Spark (PySpark/Scala) and SQL, with experience in performance tuning and troubleshooting.
• We have built and operated data platforms, including batch and streaming ingestion, transformation frameworks, orchestration, and curated data layers.
• We have experience with data lake table design and optimization across Databricks, Snowflake, or AWS, including partitioning, Z-ORDER, and file sizing.
• We are familiar with Unity Catalog or similar governance tools, including permissions, catalogs/schemas, lineage, and auditing concepts.
• We understand core software engineering practices such as Git, CI/CD, automated testing, code reviews, modular design, and documentation.
• We have implemented observability, data quality checks, and monitoring for pipelines and SQL workloads.
• We have strong communication skills and the ability to guide technical decisions across teams.
• We have experience leading effective use of approved AI-assisted software development tools, with the ability to set expectations for validating AI-generated outputs for correctness, performance, and security.
• We understand responsible AI use in engineering workflows, including data sensitivity, secure handling of inputs and outputs, resilience, security, and coaching engineers on safe, compliant adoption.
• Preferred: We have experience with Databricks Workflows, Delta Live Tables, Structured Streaming, and Databricks SQL Warehouses.
• Preferred: We have used transformation frameworks such as dbt and semantic layer patterns.
• Preferred: We have experience with infrastructure as code, such as Terraform, and automated environment provisioning.
• Preferred: We have worked in regulated-data environments with privacy controls and enterprise data governance programs.

Responsibilities:

• We define and steer the technical roadmap for our Databricks lakehouse/database platform across ingestion, storage, modeling, and serving, using clear standards and reference patterns.
• We design curated datasets, including medallion-style architecture, and apply dimensional or semantic modeling where appropriate while enforcing naming, partitioning, and performance standards.
• We build for availability and consistent performance by setting SLOs, runbooks, alerting, incident response processes, and strong operational practices for pipelines and SQL workloads.
• We implement and maintain governance, least-privilege access, auditing, data classification, and lifecycle management.
• We tune Spark and SQL workloads, optimize clusters and warehouses, manage caching and storage patterns, and apply cost observability or chargeback where needed.
• We establish engineering standards for code quality, testing, CI/CD, branching, documentation, and review, while creating reusable libraries and templates.
• We coach engineers, lead design reviews, and work with stakeholders to translate business needs into scalable data platform capabilities.
• We drive adoption of enterprise-approved AI-assisted engineering practices to improve code quality, delivery speed, and operational outcomes.
• We apply knowledge of SDLC tooling, including authorized AI-assisted development and automation capabilities, to increase the value delivered through automation.

Technologies:

• AI
• AWS
• CI/CD
• Copilot
• Databricks
• Git
• GitHub
• Support
• Marketing
• PySpark
• SQL
• Scala
• Security
• Snowflake
• Spark
• Terraform
• Unity
• dbt
• Cloud
• GameDev

More:

hackajob is collaborating with JPMorgan Chase to connect with exceptional professionals for this Lead Software Engineer opportunity within Corporate Functions, specifically Corporate - Employee Platforms. We are an agile team focused on enhancing, building, and delivering trusted, market-leading technology products in a secure, stable, and scalable way. We use AI-assisted development tools such as GitHub Copilot and native Databricks capabilities to accelerate coding, troubleshooting, and self-service analytics while maintaining strong engineering controls and review practices. JPMorganChase is one of the oldest financial institutions, serving consumers, small businesses, and some of the worlds largest corporate, institutional, and government clients. We offer a competitive total rewards package, with pay based on role, experience, skills, and location, plus eligible incentives and benefits such as health coverage, wellness centers, retirement savings, backup childcare, tuition reimbursement, mental health support, and financial coaching. We are committed to diversity, inclusion, and equal opportunity, and our Corporate Functions team supports the business across areas such as finance, risk, human resources, and marketing.

last updated 40 week of 2026