Spark Platform Engineer: Scale, Multi-Tenant & Kubernetes

Doordashusa; San Francisco, Sunnyvale, Seattle, New York (CA, CA, WA, NY); Hybrid; USD 131,000 - 192,000; Full time

Benefits offered by this job

401(k) plan with employer matching

16 weeks paid parental leave

Wellness benefits

Commuter benefits match

Paid time off and paid sick leave

Job summary

DoorDash is seeking a Software Engineer for the Spark Platform to work on our in-house Spark deployment, covering runtime upgrades, multi-tenant scheduling, and cluster lifecycle automation. You will collaborate with platform consumers across the company and relocate as needed to Bay Area, Seattle, or NYC.

You will join a team focused on running Spark at scale, with emphasis on platform operations rather than individual Spark jobs, and you will work across Kubernetes-based infrastructure and

Qualifications

• 2+ years of experience operating production distributed systems.
• Experience operating Apache Spark at scale on EMR/Databricks/in-house deployments with a focus on platform operations.
• Hands-on experience operating production systems on Kubernetes — controllers, operators, CRDs and multi-tenant clusters.
• Familiarity with batch or big-data schedulers (YuniKorn, Volcano, Kueue) and/or Spark-on-Kubernetes operator.
• Familiarity with observability stacks (Prometheus/OpenTelemetry) and defining SLOs/SLIs.
• Comfort with AWS cloud environment — VPC, instances, spot/preemptible markets, autoscaling.
• Proficiency in Python/Go/Scala/Java; SQL fluency.
• Willing to relocate or be located in Bay Area/Seattle/NYC.

Responsibilities

• Build and operate an in-house Spark platform at company-wide scale.
• Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement.
• Own cluster lifecycle automation: provisioning, upgrades, capacity changes, and node failures.
• Build observability and incident automation for end-to-end debugging and on-call efficiency.
• Collaborate with senior engineers on shuffle, runtime, and architecture work.

Skills

Apache Spark

Kubernetes

Python

Go

Scala

Java

SQL

Observability

AWS

Multi-tenant scheduling

Education

B.S./M.S./PhD in CS or equivalent

Tools

Spark on Kubernetes operator

Batch/big-data schedulers (YuniKorn/Volcano/Kueue)

Job description

DoorDash is seeking a Software Engineer for the Spark Platform to work on our in-house Spark deployment, covering runtime upgrades, multi-tenant scheduling, and cluster lifecycle automation. You will collaborate with platform consumers across the company and relocate as needed to Bay Area, Seattle, or NYC.

You will join a team focused on running Spark at scale, with emphasis on platform operations rather than individual Spark jobs, and you will work across Kubernetes-based infrastructure and