Spark Platform Engineer: Scale, Scheduling & Reliability

DoorDash USA; San Francisco (CA); Hybrid; USD 131,000 - 192,000; Full time

Benefits offered by this job

401(k) plan with employer matching

Paid parental leave

Wellness benefits

Commuter benefits match

Paid time off & paid sick leave

Medical, dental, and vision benefits

Job summary

DoorDash is seeking a Software Engineer for the Spark Platform to own and optimize our in-house Spark deployment across company workloads. You will work on runtime upgrades, multi-tenant scheduling, and Kubernetes-based cluster management while collaborating with platform consumers across the company.

The role requires hands-on experience with distributed systems at scale, Spark ecosystems (EMR/Databricks/in-house), and strong Kubernetes operations.

Qualifications

• BS/MS/PhD in Computer Science or equivalent.
• 2+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale on EMR, Databricks, or an in-house deployment with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi-tenant scheduling).
• Hands-on experience operating production systems on Kubernetes—controllers, operators, custom resources, and the failure modes that show up in multi-tenant clusters.
• Familiarity with batch or big-data schedulers (YuniKorn, Volcano, Kueue, or equivalent) and/or with the Spark-on-Kubernetes operator.
• Familiarity with observability stacks (Prometheus, OpenTelemetry, distributed tracing, structured logging) and with defining SLOs and SLIs that change team behavior.
• Comfort working in a cloud environment (AWS preferred) — VPC networking, instance lifecycle, spot/preemptible markets, and autoscaling primitives.
• Professional experience with Python, Go, Scala, or Java; SQL fluency.
• A bias toward incremental rollout, measurement, and reducing toil.
• You are located or willing to relocate to the Bay Area, Seattle, or NYC.

Responsibilities

• Build and operate an in-house Spark platform that runs at company-wide scale, spanning runtime, scheduler, reliability, and user-facing tooling.
• Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement that let a small team serve dozens of consumer teams.
• Own pieces of cluster lifecycle automation — provisioning, upgrades, capacity changes, and node-failure handling — at a scale where these stop being manual events.
• Build the observability and incident automation that make the platform debuggable end-to-end and keep on-call sustainable as the team and the workload grow.
• Partner with senior engineers on shuffle, runtime, and architecture work, and grow into deeper ownership of those areas over time.

Skills

Distributed systems

Spark

Python

SQL

Java

Education

Bachelor's/MS/PhD in Computer Science or equivalent

Tools

Kubernetes

Spark-on-Kubernetes

Prometheus

OpenTelemetry

Databricks

Amazon EMR

Job description

DoorDash is seeking a Software Engineer for the Spark Platform to own and optimize our in-house Spark deployment across company workloads. You will work on runtime upgrades, multi-tenant scheduling, and Kubernetes-based cluster management while collaborating with platform consumers across the company.

The role requires hands-on experience with distributed systems at scale, Spark ecosystems (EMR/Databricks/in-house), and strong Kubernetes operations.