Sr. Software Engineer - Data Compute Platform | Roblox NEW

The Briefed

San Mateo (CA)

On-site

USD 243,000 - 295,000

Full time

11 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Equity compensation
Healthcare
Onsite meals

Job summary

Roblox in San Mateo is hiring a Senior Software Engineer for the Data Platform Compute team to own the Spark-on-Kubernetes compute layer and the job submission APIs that power analysts, engineers, and ML workloads. You will design, operate and scale the EKS clusters, build auto-tuning and cost-control features, and partner with Data Engineering, ML, Analytics and Infra to optimize infrastructure patterns.

This role emphasizes reliability, SLO-driven observability, on-call ownership, and

Qualifications

  • 5+ years of professional experience building distributed systems.
  • Bachelor of Science in Computer Science or equivalent.
  • Hands-on experience operating Spark on Kubernetes (EKS/GKE).
  • Strong knowledge of AWS or GCP and IaC tooling.

Responsibilities

  • Design, build, and operate Spark job submission APIs and libraries.
  • Deliver Spark Connect for notebook workflows and cost-aware compute management.
  • Develop auto-tuning engines to optimize performance and spend.
  • Build lifecycle automation for EKS clusters with zero disruption.
  • Own batch scheduling and multi-tenant fairness and migrations.

Skills

Distributed systems
Backend languages
On-call ownership
Performance tuning

Education

Bachelor of Science in Computer Science

Tools

Kubernetes (EKS/GKE)
Apache Spark on Kubernetes
AWS
Terraform
Helm
Argo
Karpenter
Spark Connect

Job description

Roblox’s data platform processes petabytes of data daily, powering analytics, machine learning, and core product decisions. As a Senior Software Engineer on the Data Platform Compute team, you will build and operate the compute layer beneath this ecosystem: the Kubernetes (EKS) clusters running all Apache Spark workloads, as well as the software enabling engineers, analysts, and ML practitioners to submit and execute jobs reliably and cost-effectively. You will own the systems sitting between internal users and the cluster—including Spark job submission APIs, internal libraries, interactive notebook compute (Spark Connect), auto-tuning engines, and cluster lifecycle automation.

Key Responsibilities

Compute Services & Interactive Infrastructure

  • Core Compute Platform Services: Design, build, and operate the software backing Spark job submission, including API design, scheduling integrations, and shared user libraries.
  • Interactive Spark at Scale: Deliver Spark Connect to power notebook workflows, provisioning and managing EKS compute infrastructure with robust isolation, quotas, and cost controls.
  • Auto-Tuning Engine Development: Develop automated job tuners that analyze Spark history and event logs to recommend or apply configuration changes that optimize performance and compute spend.

Cluster Automation & Batch Scheduling

  • EKS Fleet Management Automation: Build lifecycle automation for EKS clusters, executing blue-green rollouts to validate new AMIs and Kubernetes versions with zero disruption to active workloads.
  • Batch Scheduling Ownership: Operate and upgrade YuniKorn-based batch scheduling, overseeing queue architectures, tenant quotas, fairness policies, and version migrations.
  • Emerging Technology Evaluation: Evaluate next-generation compute technologies (e.g., native Spark acceleration via Apache Comet), executing benchmarks on real petabyte-scale workloads to drive adoption.
  • Reliability & Cost Optimization: Define SLOs, build observability pipelines, and reduce operational toil, incidents, and platform spend across multi-tenant clusters.
  • Cross-Functional Alignment: Partner with Data Engineering, ML, Analytics, and Infrastructure teams to streamline workflows and migrate users to optimized infrastructure patterns.
Requirements

Technical Depth & Platform Engineering Foundations

  • Experience Track: 5+ years of professional experience building and operating production-grade distributed systems or core infrastructure platforms.
  • Education: Bachelor of Science in Computer Science or equivalent practical experience.
  • Spark on Kubernetes Expertise: Hands-on experience operating Apache Spark on Kubernetes (EKS, GKE, or self-managed) at the platform layer—including driver/executor behavior, resource tuning, and debugging failed or slow jobs.
  • Kubernetes Depth: Strong operational experience with production Kubernetes clusters at scale (cluster upgrades, node lifecycle management, autoscaling, operators/controllers).
  • Backend Language Proficiency: Strong proficiency in Go, Java, Scala, or Clojure with a track record of writing clean, maintainable backend code (Python proficiency is a plus for tuning tools).
  • Cloud Infrastructure: Experience with AWS (EKS, EC2, IAM, S3) or GCP equivalents, alongside Infrastructure-as-Code (IaC).
  • Operational Ownership: Experience taking systems from architecture through production, including on-call rotations, incident response, and SLO/metric observability.
Preferred Qualifications
  • Experience with batch schedulers on Kubernetes (YuniKorn, Volcano, Kueue).
  • Experience with interactive and gateway layers (Spark Connect, Livy, Kyuubi).
  • Experience with Spark Operator, Karpenter, Helm, Terraform, or Argo.
  • Experience building automated tooling using Spark event logs/metrics for cost attribution, tuning, or anomaly detection.
  • Track record of maintaining internal Spark builds or executing Spark version upgrades.
  • Familiarity with native execution engines (Comet, Gluten, Velox) or JVM performance tuning.
  • Experience migrating legacy data workloads from YARN/Hadoop to Kubernetes.
  • Active open-source contributions to the Apache Spark or Kubernetes ecosystems.
  • Experience operating platforms at consumer-internet scale (100M+ users).
Total Rewards & Benefits
  • Compensation & Equity: San Mateo base salary range ($243,290 – $295,250 USD) + competitive Roblox equity compensation.
  • Healthcare & Wellbeing: Comprehensive health, dental, and vision insurance packages for employees and eligible dependents.
  • Onsite Perks & Campus: Modern San Mateo headquarters featuring daily catered meals, team events, and state-of-the-art technical setups.
  • Time Off & Retirement: Generous vacation policies, paid company holidays, paid parental leave, and 401(k) matching programs.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Software Engineer (Data Infrastructure Compute)
Senior Software Engineer (Data Infrastructure Compute)

Roblox Corporation • San Mateo (CA)

On-site
USD 243,000 - 295,000
Senior Software Engineer, Data Infrastructure Compute
Senior Software Engineer, Data Infrastructure Compute

Showcify, Inc. • San Mateo (CA)

On-site
USD 243,000 - 295,000
Equity compensation
Senior Software Engineer, Data Infrastructure Compute
Senior Software Engineer, Data Infrastructure Compute

Roblox Corporation • San Mateo (CA)

On-site
USD 243,000 - 295,000
Principal Cloud & Kubernetes Engineer - Compute NEW
Principal Cloud & Kubernetes Engineer - Compute NEW

The Briefed • San Mateo (CA)

On-site
USD 345,000 - 399,000
Daily catered meals
Team events
State-of-the-art technical setups
+3
Senior Software Engineer - Consumer Apps Platform NEW
Senior Software Engineer - Consumer Apps Platform NEW

The Briefed • San Mateo (CA)

On-site
USD 197,000 - 243,000
Equity grants
Healthcare coverage
Onsite meals & facilities
Senior Software Engineer, Data Insights
Senior Software Engineer, Data Insights

Roblox Corporation • San Mateo (CA)

On-site
USD 196,750 - 243,290
Senior Data Compute Platform Engineer (Spark on Kubernetes)
Senior Data Compute Platform Engineer (Spark on Kubernetes)

Roblox Corporation • San Mateo (CA)

On-site
USD 243,000 - 295,000
Senior Data Infra Engineer — Spark on Kubernetes
Senior Data Infra Engineer — Spark on Kubernetes

Showcify, Inc. • San Mateo (CA)

On-site
USD 243,000 - 295,000
Equity compensation
Senior Data Compute Platform Engineer (Spark on Kubernetes)
Senior Data Compute Platform Engineer (Spark on Kubernetes)

The Briefed • San Mateo (CA)

On-site
USD 243,000 - 295,000
Equity compensation
Healthcare
Onsite meals
Senior Distributed Systems Engineer - Storage NEW
Senior Distributed Systems Engineer - Storage NEW

The Briefed • San Mateo (CA)

On-site
USD 243,000 - 295,000
Competitive salary
Roblox equity
Health insurance
+3