Senior Software Engineer, Spark Platform

DoorDash; New York (NY); Hybrid; USD 193,800 - 285,000; Full time

Benefits offered by this job

401(k) plan with employer matching

Paid parental leave

Wellness benefits

Paid time off

Job summary

DoorDash is looking for a Senior Software Engineer to lead architectural decisions for their in-house Spark deployment. This role focuses on distributed systems challenges, particularly while working with Apache Spark. The position emphasizes technical leadership within a fast-paced, collaborative environment and offers a hybrid work model based in NYC.

Ideal candidates should possess extensive experience with distributed systems, Apache Spark, and Kubernetes. Comprehensive benefits and competitive compensation packages are offered.

Qualifications

• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep experience with Apache Spark internals and platform operations.
• Fluency operating workloads on Kubernetes in production.

Responsibilities

• Set the technical direction for Spark-on-Kubernetes platform.
• Own distributed-systems problems like multi-tenant scheduling.
• Partner with Engineering Manager on technical roadmap and hiring.

Skills

Distributed systems

Apache Spark

Kubernetes

Scala

Java

Python

SQL

Education

B.S., M.S., or Ph.D. in Computer Science

Tools

Amazon EMR

Databricks

Celeborn

YuniKorn

Job description

About the Team

The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem—execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve. Orchestrating and operating thousands of Spark cluster deployments is a complex distributed system problem which the team invests heavily in runtime optimization, systems architecture, multi‑tenant scheduling, and end‑user tooling.

About the Role

As a Senior Software Engineer on Spark Platform, you will set the technical direction for our in‑house Spark deployment and shape the architecture that will run DoorDash's data, analytics, and ML compute for the next five years and beyond. You will own the deep, cross‑cutting problems that span the runtime, the shuffle service, the scheduler, and the overall service reliability—making the architectural calls that compound across the platform's lifetime. You will partner with the Engineering Manager on technical roadmap, hiring, and team shape, and act as the senior technical voice in cross‑team partnerships with Data Engineering, ML Platform, and product engineering teams that depend on the platform.

Location: Candidates must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team.

Responsibilities

• Set the multi‑year technical direction for an in‑house Spark‑on‑Kubernetes platform—runtime, shuffle, scheduler, reliability—and make the architectural calls that compound for years.
• Own the deepest distributed‑systems problems on the team: shuffle architecture, multi‑tenant scheduling, runtime performance, and the failure modes that only show up at scale.
• Partner with the Engineering Manager on technical roadmap, hiring, interview design, and team shape as the team continues to grow.
• Uplevel the rest of the team through design reviews, mentorship, and raising the bar on what we ship.
• Represent Spark Platform in cross‑team architecture forums and shape how data, analytics, and ML workloads land on the platform.

Requirements

• B.S., M.S., or Ph.D. in Computer Science or equivalent.
• 6+ years of industry experience designing and operating distributed systems at scale.
• Deep, hands‑on experience with Apache Spark—internals, query execution, shuffle, the executor/driver model—at platform scale on Amazon EMR, Databricks, or an in‑house deployment, with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi‑tenant scheduling) rather than authoring individual Spark jobs.
• Production experience with one or more of: remote/external shuffle systems (Celeborn, Magnet, Cosco, or similar), batch/big‑data schedulers (YuniKorn, Volcano, Kueue, or the Spark‑on‑Kubernetes operator), or the observability and SRE patterns that make distributed compute platforms operable.
• Strong fluency operating workloads on Kubernetes in production—operator patterns, executor pod lifecycle, network topology, and the multi‑tenant failure modes that show up at scale.
• Familiarity with data lake table formats such as Apache Iceberg or Delta Lake, and with the query and SQL engines that read them.
• Track record of acting as a technical leader on a platform team—setting direction, mentoring, and partnering with management on roadmap and hiring.
• Professional experience with Scala, Java, Python, or Go; strong SQL.
• You are located or willing to relocate to the Bay Area, Seattle, or NYC.

Compensation

Base salary ranges are localized by location: Bay Area $130,600–$192,000 USD; Seattle $159,800–$235,000 USD; NYC $193,800–$285,000 USD. In addition to base salary, equity grants may be awarded. The role includes a comprehensive benefits package covering a 401(k) plan with employer matching, paid parental leave, wellness benefits, commuter benefits, paid time off, paid sick leave, medical, dental, vision, 11 paid holidays, disability, basic life insurance, family‑forming assistance, and a mental health program.

EEO Statement

Statement of Non‑Discrimination: In keeping with our beliefs and goals, no employee or applicant will face discrimination or harassment based on race, color, ancestry, national origin, religion, age, gender, marital/domestic partner status, sexual orientation, gender identity or expression, disability status, or veteran status. Above and beyond discrimination and harassment based on “protected categories,” we also strive to prevent other subtler forms of inappropriate behavior (i.e., stereotyping) from ever gaining a foothold in our office. Whether blatant or hidden, barriers to success have no place at DoorDash. We value a diverse workforce—people who identify as women, non‑binary or gender non‑conforming, LGBTQIA+, American Indian or Native Alaskan, Black or African American, Hispanic or Latinx, Native Hawaiian or Other Pacific Islander, differently‑abled, caretakers and parents, and veterans are strongly encouraged to apply.