Senior Spark Platform Engineer — Scale & Sharding Leader

DoorDash; New York (NY); Hybrid; USD 193,800 - 285,000; Full time

Benefits offered by this job

401(k) plan with employer matching

Paid parental leave

Wellness benefits

Paid time off

Job summary

DoorDash is looking for a Senior Software Engineer to lead architectural decisions for their in-house Spark deployment. This role focuses on distributed systems challenges, particularly while working with Apache Spark. The position emphasizes technical leadership within a fast-paced, collaborative environment and offers a hybrid work model based in NYC.

Ideal candidates should possess extensive experience with distributed systems, Apache Spark, and Kubernetes. Comprehensive benefits and competitive compensation packages are offered.

Qualifications

• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep experience with Apache Spark internals and platform operations.
• Fluency operating workloads on Kubernetes in production.

Responsibilities

• Set the technical direction for Spark-on-Kubernetes platform.
• Own distributed-systems problems like multi-tenant scheduling.
• Partner with Engineering Manager on technical roadmap and hiring.

Skills

Distributed systems

Apache Spark

Kubernetes

Scala

Java

Python

SQL

Education

B.S., M.S., or Ph.D. in Computer Science

Tools

Amazon EMR

Databricks

Celeborn

YuniKorn

Job description

DoorDash is looking for a Senior Software Engineer to lead architectural decisions for their in-house Spark deployment. This role focuses on distributed systems challenges, particularly while working with Apache Spark. The position emphasizes technical leadership within a fast-paced, collaborative environment and offers a hybrid work model based in NYC.

Ideal candidates should possess extensive experience with distributed systems, Apache Spark, and Kubernetes. Comprehensive benefits and competitive compensation packages are offered.