Spark Platform Engineer: Scale, Multi-Tenant Kubernetes

DoorDash, Inc.; New York (NY); On-site; USD 131,000 - 192,000; Full time

Benefits offered by this job

Health benefits

401(k) matching

Paid parental leave

Paid time off

Paid sick leave

Medical, dental, and vision coverage

Wellness benefits

Commuter benefits

Job summary

DoorDash's Spark Platform team builds and operates its in-house Spark ecosystem, handling runtime upgrades, multi-tenant scheduling, and end-to-end observability for data, analytics, and ML workloads at scale.

As a Software Engineer, you will work across surfaces of the deployment, partnering with platform consumers, and help automate cluster lifecycle, bin-packing on Kubernetes, and reliability tooling, reporting to the Engineering Manager.

Qualifications

• BS/MS/PhD in Computer Science or equivalent.
• 2+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale (in-house, EMR, or Databricks).
• Hands-on experience operating production systems on Kubernetes (controllers and operators).
• Familiarity with observability stacks (Prometheus/OpenTelemetry) and SLO/SLI definition.
• Comfort in cloud environments (AWS).
• Proficiency in Python, Go, Scala, or Java; SQL fluency.

Responsibilities

• Build and operate an in-house Spark platform at company-wide scale across runtime, scheduler, reliability, and user-facing tooling.
• Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement for many consumer teams.
• Own cluster lifecycle automation including provisioning, upgrades, and node-failure handling.
• Develop observability and incident automation to maintain reliable, scalable operations.
• Collaborate with senior engineers on shuffle, runtime, and architecture work and grow ownership over time.

Skills

Distributed systems operations

Apache Spark

Kubernetes

Python

Go

Scala

Java

AWS

Observability

SQL

Education

BS/MS/PhD in Computer Science or equivalent

Tools

Spark on Kubernetes operator

Prometheus

OpenTelemetry

Databricks

EMR

Job description

DoorDash's Spark Platform team builds and operates its in-house Spark ecosystem, handling runtime upgrades, multi-tenant scheduling, and end-to-end observability for data, analytics, and ML workloads at scale.

As a Software Engineer, you will work across surfaces of the deployment, partnering with platform consumers, and help automate cluster lifecycle, bin-packing on Kubernetes, and reliability tooling, reporting to the Engineering Manager.