Software Engineer, Spark Platform

DoorDash; New York (NY); Hybrid; USD 130,600 - 192,000; Full time

Benefits offered by this job

401(k) plan with employer matching

16 weeks of paid parental leave

wellness benefits

paid time off

medical, dental, and vision benefits

Job summary

DoorDash is hiring a Software Engineer to work on the Spark Platform team, focusing on the company's Apache Spark ecosystem. This hybrid role involves responsibilities such as managing cluster automation, multi-tenant scheduling, and ensuring the platform’s observability and efficiency.

The ideal candidate will have significant experience with Apache Spark at scale, Kubernetes, and a strong educational background in Computer Science. The position is located in New York and offers a competitive salary and comprehensive benefits package.

Qualifications

• 24+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale with a focus on platform operations.
• Hands-on experience operating production systems on Kubernetes.

Responsibilities

• Build and operate an in-house Spark platform that runs at company-wide scale.
• Drive multi-tenant scheduling and executor bin-packing.
• Own pieces of cluster lifecycle automation.

Skills

Operating production distributed systems

Apache Spark

Kubernetes

Python

SQL fluency

Education

B.S., M.S., or Ph.D. in Computer Science or equivalent

Tools

Amazon EMR

Databricks

Prometheus

OpenTelemetry

Job description

About the Team

The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem — the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve. Orchestrating and operating thousands of Spark cluster deployments is a complex distributed system problem which the team invests heavily in runtime optimization, systems architecture, multi-tenant scheduling, and end‑user tooling.

About the Role

As a Software Engineer on Spark Platform, you will execute across the surfaces of our in‑house Spark deployment that serves the entire company. The work spans Spark runtime upgrades and performance, multi‑tenant scheduling and executor bin‑packing on Kubernetes, cluster lifecycle automation, and the observability and incident automation that keep the platform sustainable. You will move between layers as the work demands — picking up the next high‑leverage problem regardless of where it sits — and partner closely with the rest of the team and with platform consumers across the company.

You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team.

Responsibilities

• Build and operate an in‑house Spark platform that runs at company‑wide scale, spanning runtime, scheduler, reliability, and user‑facing tooling.
• Drive multi‑tenant scheduling, executor bin‑packing, and cost‑aware placement that let a small team serve dozens of consumer teams.
• Own pieces of cluster lifecycle automation — provisioning, upgrades, capacity changes, and node‑failure handling — at a scale where these stop being manual events.
• Build the observability and incident automation that make the platform debuggable end‑to‑end and keep on‑call sustainable as the team and the workload grow.
• Partner with senior engineers on shuffle, runtime, and architecture work, and grow into deeper ownership of those areas over time.

Qualifications

• B.S., M.S., or Ph.D. in Computer Science or equivalent.
• 24+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale on Amazon EMR, Databricks, or an in‑house deployment — with a focus on platform operations (runtime upgrades, cluster lifecycle, shuffle, observability, multi‑tenant scheduling) rather than authoring individual Spark jobs.
• Hands‑on experience operating production systems on Kubernetes — controllers, operators, custom resources, and the failure modes that show up in multi‑tenant clusters.
• Familiarity with batch or big‑data schedulers (YuniKorn, Volcano, Kueue, or equivalent) and/or with the Spark‑on‑Kubernetes operator.
• Familiarity with observability stacks (Prometheus, OpenTelemetry, distributed tracing, structured logging) and with defining SLOs and SLIs that change team behavior.
• Comfort working in a cloud environment (AWS preferred) — VPC networking, instance lifecycle, spot/preemptible markets, and autoscaling primitives.
• Professional experience with Python, Go, Scala, or Java; SQL fluency.
• A bias toward incremental rollout, measurement, and reducing toil.
• Located or willing to relocate to the Bay Area, Seattle, or NYC.

Compensation & Benefits

The successful candidate’s starting pay will fall within a pay range based on skills, experience, qualifications, location, and market conditions. Base salary is localized. Pay ranges are as follows:

• $130,600 – $192,000 USD (San Francisco)
• $159,800 – $235,000 USD (Sunnyvale)
• $193,800 – $285,000 USD (Seattle)

In addition to base salary, the compensation for this role includes opportunities for equity grants. Talk to your recruiter for more information.

DoorDash offers a comprehensive benefits package to all regular employees, including a 401(k) plan with employer matching, 16 weeks of paid parental leave, wellness benefits, commuter benefits match, paid time off and paid sick leave in compliance with applicable laws, medical, dental, and vision benefits, 11 paid holidays, disability and basic life insurance, family‑forming assistance, and a mental health program.

Our Commitment to Diversity and Inclusion

We’re committed to growing and empowering a more inclusive community within our company, industry, and cities. That’s why we hire and cultivate diverse teams of people from all backgrounds, experiences, and perspectives. We believe that true innovation happens when everyone has room at the table and the tools, resources, and opportunity to excel.

Statement of Non‑Discrimination: In keeping with our beliefs and goals, no employee or applicant will face discrimination or harassment based on race, color, ancestry, national origin, religion, age, gender, marital/domestic partner status, sexual orientation, gender identity or expression, disability status, or veteran status. Above and beyond discrimination and harassment based on “protected categories,” we also strive to prevent other subtler forms of inappropriate behavior (i.e., stereotyping) from ever gaining a foothold in our office. Whether blatant or hidden, barriers to success have no place at DoorDash. We value a diverse workforce and strongly encourage applicants who identify as women, non‑binary or gender non‑conforming, LGBTQIA+, American Indian or Native Alaskan, Black or African American, Hispanic or Latinx, Native Hawaiian or Other Pacific Islander, differently‑abled, caretakers and parents, and veterans to apply.

Pursuant to the San Francisco Fair Chance Ordinance, Los Angeles Fair Chance Initiative for Hiring Ordinance, and any other state or local hiring regulations, we will consider for employment any qualified applicant, including those with arrest and conviction records, in a manner consistent with the applicable regulation.

If you need any accommodations, please inform your recruiting contact upon initial connection.