Senior Spark Platform Engineer — Scale & Reliability

Doordashusa; San Francisco, Sunnyvale, Seattle, New York (CA, CA, WA, NY); Hybrid; USD 160.000 - 235.000; Vollzeit

Benefits dieser Stelle

401(k) with matching

16 weeks paid parental leave

Wellness benefits

Commuter benefits match

Paid time off & paid sick leave

Medical/dental/vision benefits

11 paid holidays

Disability and basic life insurance

Family-forming assistance

Mental health program

Zusammenfassung

DoorDash is seeking a Senior Software Engineer for the Spark Platform to set the technical direction of our in-house Spark deployment and shape the architecture for data, analytics, and ML workloads. You will own cross-cutting runtime, shuffle, and scheduler challenges, partnering with the Engineering Manager on roadmaps and team growth.

You will lead architectural decisions, mentor engineers, and collaborate with Data Engineering/ML Platform teams to scale our platform for years to come.

Qualifikationen

• 6+ years designing and operating distributed systems at scale.
• Deep Spark internals, query execution, shuffle, and platform-scale experience (EMR/Databricks/in-house).
• Production experience with remote shuffle systems or batch/big-data schedulers or observability patterns for distributed platforms.
• Strong fluency with Kubernetes production patterns, operator models, and multi-tenant failure modes.
• Familiarity with Iceberg or Delta Lake formats and related SQL engines.
• Track record as a technical leader on a platform team with roadmap/hiring collaboration.
• Proficiency in Scala, Java, Python, or Go and strong SQL.
• Willing to relocate to the Bay Area, Seattle, or NYC.

Aufgaben

• Set the multi-year technical direction for an in-house Spark-on-Kubernetes platform — runtime, shuffle, scheduler, reliability.
• Own the deepest distributed-systems problems at scale: shuffle, multi-tenant scheduling, runtime performance, failure modes.
• Partner with the Engineering Manager on roadmap, hiring, interview design, and team shaping as the team grows.
• Uplevel the team through design reviews, mentorship, and raising the bar on what we ship.
• Represent Spark Platform in cross-team forums on data, analytics, and ML workloads.

Kenntnisse

Apache Spark

Distributed systems

Kubernetes in production

Scala/Java/Python/Go

SQL proficiency

Ausbildung

B.S./M.S./PhD in Computer Science

Tools

Kubernetes

YuniKorn/Volcano/Kueue

Iceberg/Delta Lake

Spark-on-Kubernetes operator

Jobbeschreibung

DoorDash is seeking a Senior Software Engineer for the Spark Platform to set the technical direction of our in-house Spark deployment and shape the architecture for data, analytics, and ML workloads. You will own cross-cutting runtime, shuffle, and scheduler challenges, partnering with the Engineering Manager on roadmaps and team growth.

You will lead architectural decisions, mentor engineers, and collaborate with Data Engineering/ML Platform teams to scale our platform for years to come.