Senior Spark Platform Engineer: Scale & Reliability

DoorDash, Inc.; Seattle (WA); Hybrid; USD 160,000 - 235,000; Full time

Benefits offered by this job

401(k) with employer matching

16 weeks paid parental leave

Wellness benefits

Commuter benefits match

Paid time off & paid sick leave

Medical, dental, vision benefits

Job summary

DoorDash's Spark Platform team is seeking a Senior Software Engineer to define the in-house Spark deployment, shaping runtime, shuffle, and scheduler architecture for data, analytics, and ML workloads. The role focuses on building a scalable, reliable platform that powers edges of Dash's data stack.

You will tackle deep distributed-systems challenges at scale, partner with engineering leadership on roadmaps, and mentor teams while collaborating across Data Eng, ML Platform, and product

Qualifications

• Bachelor-level degree in CS or equivalent; advanced degrees preferred.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep, hands-on experience with Apache Spark internals and runtime.
• Experience with multi-tenant environments and platform operations.

Responsibilities

• Set the multi-year technical direction for the in-house Spark platform.
• Own deep distributed-systems problems: shuffle, multi-tenant scheduling, runtime performance.
• Collaborate with Engineering Manager on roadmap, hiring, and team structure.
• Mentor teammates and raise the bar on what is shipped.
• Represent Spark Platform in cross-team architecture forums.

Skills

Distributed systems

Apache Spark

Kubernetes production

Scala/Java/Python/Go

SQL

Education

B.S./M.S./PhD in CS or equivalent

Tools

Celeborn/Shuffle

YuniKorn/Volcano/Kueue

Spark-on-Kubernetes operator

Job description

DoorDash's Spark Platform team is seeking a Senior Software Engineer to define the in-house Spark deployment, shaping runtime, shuffle, and scheduler architecture for data, analytics, and ML workloads. The role focuses on building a scalable, reliable platform that powers edges of Dash's data stack.

You will tackle deep distributed-systems challenges at scale, partner with engineering leadership on roadmaps, and mentor teams while collaborating across Data Eng, ML Platform, and product