Spark Platform Engineer — Scale, Scheduler & Kubernetes

DoorDash, Inc.; Sunnyvale (CA); Hybrid; USD 131,000 - 192,000; Full time

Benefits offered by this job

401(k) plan with matching

16 weeks of paid parental leave

Wellness benefits

Commuter benefits match

Paid time off

Medical/dental/vision benefits

Job summary

DoorDash, Inc. is hiring a Software Engineer for the Spark Platform to operate and extend our in‑house Spark deployment at company scale. You will work across runtime upgrades, multi‑tenant scheduling, and cluster lifecycle automation, collaborating with platform consumers across the company.

You’ll partner with senior engineers on shuffle, runtime, and architecture, and grow ownership of core platform areas while supporting a hybrid, location‑flexible role in the Bay Area, NYC, or Seattle.

Qualifications

• 2+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale on cloud or in‑house deployments.
• Hands‑on experience operating production systems on Kubernetes.
• Familiarity with batch schedulers (YuniKorn, Volcano, Kueue) or Spark‑on‑Kubernetes.
• Familiarity with observability stacks and defining SLOs/SLIs.
• Proficiency in Python, Go, Scala, or Java; SQL fluency.
• Willing to relocate to the Bay Area, Seattle, or NYC.
• B.S./M.S./PhD in Computer Science or equivalent.

Responsibilities

• Build and operate an in‑house Spark platform at company‑wide scale across runtime, scheduler, reliability, and tooling.
• Drive multi‑tenant scheduling, executor bin‑packing, and cost‑aware placement for many consumer teams.
• Own cluster lifecycle automation: provisioning, upgrades, capacity changes, and node failures.
• Build observability and incident automation for end‑to‑end debugging and scalable on‑call practices.
• Partner with senior engineers on shuffle, runtime, and architecture work; grow ownership over time.

Skills

Distributed systems

Apache Spark

Kubernetes

Batch schedulers

Observability

Cloud AWS

Programming: Python/Go/Scala/Java

SQL proficiency

Relocation willingness

Education

B.S./M.S./PhD in CS

Tools

Prometheus

OpenTelemetry

Job description

DoorDash, Inc. is hiring a Software Engineer for the Spark Platform to operate and extend our in‑house Spark deployment at company scale. You will work across runtime upgrades, multi‑tenant scheduling, and cluster lifecycle automation, collaborating with platform consumers across the company.

You’ll partner with senior engineers on shuffle, runtime, and architecture, and grow ownership of core platform areas while supporting a hybrid, location‑flexible role in the Bay Area, NYC, or Seattle.