Senior Data Platform Engineer - Spark on Kubernetes Scale

Roblox Corporation; San Mateo (CA); On-site; USD 243,000 - 295,000; Full time

Job summary

Roblox Corporation seeks a Senior Software Engineer for Data Infrastructure Compute to build and operate the compute layer for Spark workloads on Kubernetes. You will own the job submission service, interactive notebook compute, and multi-tenant cluster management, influencing design choices and driving reliability at scale.

You will work across data engineering, ML, and analytics teams to shape the platform, define SLOs, reduce toil, and optimize cost while delivering robust, scalable compute

Qualifications

• 5+ years of experience building and operating production-grade distributed systems or infrastructure platforms.
• Hands-on experience running Apache Spark on Kubernetes (EKS, GKE, or self-managed).
• Strong experience operating production Kubernetes clusters at scale with upgrades and autoscaling.
• Proficiency in Go, Java, Scala, or Clojure and backend code quality.
• Cloud fluency with AWS or GCP and infrastructure-as-code.
• Experience taking a system from design through production operation including on-call and observability.
• Technical leadership to influence design decisions and mentor engineers.

Responsibilities

• Build and own core compute platform services behind Spark job submission, APIs, scheduling, and libraries.
• Launch interactive Spark at scale with Spark Connect and manage EKS-backed compute with isolation and cost controls.
• Run a safe, automated Kubernetes fleet with blue-green rollouts and verified AMIs.
• Develop auto-tuner for Spark to optimize performance and reduce costs.
• Own batch scheduling on Kubernetes, including YuniKorn queues, quotas, and fairness.
• Evaluate new tech (e.g., native Spark acceleration) and benchmark Roblox workloads.
• Define SLOs, improve observability, and reduce toil across a multi-tenant platform.
• Collaborate with data engineering, ML, analytics, and infrastructure teams to migrate users to better paths.

Skills

Kubernetes

Spark

Go

Java

Scala

Clojure

Python

Cloud (AWS/GCP)

Distributed systems

Technical leadership

Education

BS in Computer Science or equivalent practical experience

Tools

Spark Connect

Livy

Kyuubi

Spark Operator

Karpenter

Helm

Terraform

Argo

Job description

Roblox Corporation seeks a Senior Software Engineer for Data Infrastructure Compute to build and operate the compute layer for Spark workloads on Kubernetes. You will own the job submission service, interactive notebook compute, and multi-tenant cluster management, influencing design choices and driving reliability at scale.

You will work across data engineering, ML, and analytics teams to shape the platform, define SLOs, reduce toil, and optimize cost while delivering robust, scalable compute