Senior Spark Platform Engineer — Distributed Systems Lead

DoorDash USA; San Francisco (CA); Ibrido; USD 160.000 - 235.000; Tempo pieno

Descrizione del lavoro

DoorDash is seeking a Senior Software Engineer for the Spark Platform in the San Francisco Bay Area. You will set the technical direction for our in-house Spark deployment, shaping architecture that runs data, analytics, and ML workloads at scale.

You’ll own runtime, shuffle, scheduler, and reliability across the platform and collaborate with Data Eng, ML Platform, and product teams. You will partner with the Engineering Manager on roadmap, hiring, and team structure, mentoring others and

Competenze

• 6+ years designing and operating distributed systems at scale.
• Experience with Apache Spark internals, shuffle, executor/driver model at platform scale.
• Strong SQL and programming in Scala/Java/Python/Go.

Mansioni

• Set multi-year technical direction for in-house Spark deployment.
• Own deep distributed-systems problems across runtime, shuffle, scheduler, and reliability.
• Partner with Engineering Manager on roadmap, hiring, and team shape.
• Mentor and raise the bar through design reviews and guidance.
• Represent Spark Platform in cross-team forums and align data/ML workloads.

Conoscenze

Distributed systems design

Scala/Java/Python/Go

SQL

Kubernetes

Spark internals

Mentoring

Cross-team collaboration

Formazione

B.S./M.S./PhD in CS or equivalent

Strumenti

Apache Spark

Kubernetes

AWS EMR/Databricks

Spark-on-Kubernetes operator

Celeborn / Magnet / Cosco

Descrizione del lavoro

DoorDash is seeking a Senior Software Engineer for the Spark Platform in the San Francisco Bay Area. You will set the technical direction for our in-house Spark deployment, shaping architecture that runs data, analytics, and ML workloads at scale.

You’ll own runtime, shuffle, scheduler, and reliability across the platform and collaborate with Data Eng, ML Platform, and product teams. You will partner with the Engineering Manager on roadmap, hiring, and team structure, mentoring others and