Senior Spark Platform Engineer — Drive Distributed Systems

DoorDash, Inc.; San Francisco (CA); Hybrid; USD 160,000 - 235,000; Full time

Benefits offered by this job

401(k) with employer matching

Parental leave (16 weeks)

Paid time off

Paid sick leave

Health, dental, vision

Holidays

Disability & life insurance

Mental health program

Commuter benefits match

Wellness benefits

Job summary

DoorDash, Inc. is seeking a Senior Software Engineer for the Spark Platform to steer the in-house Spark deployment, runtimes, and reliability at scale. You will shape architecture and collaborate with Data Engineering and ML teams while relocating as needed. The role is hybrid in SF/NYC/Seattle area.

You will lead cross-team technical discussions, mentor peers, and define long-term platform direction, with responsibilities spanning runtime upgrades, shuffle, and cluster lifecycle.

Qualifications

• Experience designing and operating distributed systems at scale.
• Hands-on Apache Spark internals, shuffle, and runtime concepts.
• Production experience with Kubernetes in data platforms.

Responsibilities

• Set multi-year technical direction for in-house Spark-on-Kubernetes platform.
• Own deep distributed-systems problems including shuffle, multi-tenant scheduling, and reliability.
• Collaborate on roadmap, hiring, and cross-team partnerships.

Skills

Apache Spark

Kubernetes operations

Distributed systems

Scala/Java/Python/Go

SQL

Education

BS/MS/PhD in Computer Science or equivalent

Tools

YuniKorn

Volcano

Kueue

Job description

DoorDash, Inc. is seeking a Senior Software Engineer for the Spark Platform to steer the in-house Spark deployment, runtimes, and reliability at scale. You will shape architecture and collaborate with Data Engineering and ML teams while relocating as needed. The role is hybrid in SF/NYC/Seattle area.

You will lead cross-team technical discussions, mentor peers, and define long-term platform direction, with responsibilities spanning runtime upgrades, shuffle, and cluster lifecycle.