Senior Spark Platform Engineer — Scale & Modernize Spark

DoorDash, Inc.; New York (NY); Hybrid; USD 160,000 - 235,000; Full time

Job summary

DoorDash's Spark Platform team operates the in-house Spark deployment, runtime, shuffle service, and scheduler, powering data, analytics, and ML workloads at scale. As a Senior Software Engineer you will set the technical direction for the platform and own cross-cutting problems spanning runtime, shuffle, scheduler, and reliability.

You will partner with the Engineering Manager on roadmap and hiring, mentor teammates, and collaborate with Data Engineering, ML Platform, and product teams to land

Qualifications

• 6+ years designing and operating distributed systems at scale.
• Deep, hands-on Apache Spark experience — internals, shuffle, execution.
• Production experience with Kubernetes in large-scale environments.
• Proficiency in Scala, Java, Python, or Go; strong SQL.
• Experience with data lake formats like Iceberg/Delta is a plus.

Responsibilities

• Set multi-year technical direction for an in-house Spark deployment.
• Own deep distributed-systems problems: runtime, shuffle, scheduler, reliability.
• Partner with Engineering Manager on roadmap, hiring, and team shape.
• Mentor teammates and lead cross-team architectural discussions.
• Represent Spark Platform in cross-team forums and drive platform adoption.

Skills

Distributed systems

Apache Spark

Kubernetes production

Scala

Java

Python

Go

SQL

Education

B.S., M.S., or PhD in CS

Tools

Spark-on-Kubernetes

Remote shuffle systems

Observability & SRE patterns

Job description

DoorDash's Spark Platform team operates the in-house Spark deployment, runtime, shuffle service, and scheduler, powering data, analytics, and ML workloads at scale. As a Senior Software Engineer you will set the technical direction for the platform and own cross-cutting problems spanning runtime, shuffle, scheduler, and reliability.

You will partner with the Engineering Manager on roadmap and hiring, mentor teammates, and collaborate with Data Engineering, ML Platform, and product teams to land