Spark Platform Engineer - Scale, Kubernetes, Multi-Tenant

DoorDash; New York (NY); Hybrid; USD 130,600 - 192,000; Full time

Benefits offered by this job

401(k) plan with employer matching

16 weeks of paid parental leave

wellness benefits

paid time off

medical, dental, and vision benefits

Job summary

DoorDash is hiring a Software Engineer to work on the Spark Platform team, focusing on the company's Apache Spark ecosystem. This hybrid role involves responsibilities such as managing cluster automation, multi-tenant scheduling, and ensuring the platform’s observability and efficiency.

The ideal candidate will have significant experience with Apache Spark at scale, Kubernetes, and a strong educational background in Computer Science. The position is located in New York and offers a competitive salary and comprehensive benefits package.

Qualifications

• 24+ years of industry experience operating production distributed systems.
• Experience operating Apache Spark at scale with a focus on platform operations.
• Hands-on experience operating production systems on Kubernetes.

Responsibilities

• Build and operate an in-house Spark platform that runs at company-wide scale.
• Drive multi-tenant scheduling and executor bin-packing.
• Own pieces of cluster lifecycle automation.

Skills

Operating production distributed systems

Apache Spark

Kubernetes

Python

SQL fluency

Education

B.S., M.S., or Ph.D. in Computer Science or equivalent

Tools

Amazon EMR

Databricks

Prometheus

OpenTelemetry

Job description

DoorDash is hiring a Software Engineer to work on the Spark Platform team, focusing on the company's Apache Spark ecosystem. This hybrid role involves responsibilities such as managing cluster automation, multi-tenant scheduling, and ensuring the platform’s observability and efficiency.

The ideal candidate will have significant experience with Apache Spark at scale, Kubernetes, and a strong educational background in Computer Science. The position is located in New York and offers a competitive salary and comprehensive benefits package.