Senior Distributed Systems Engineer, Scalable GPU Platform

Thinking Machines Lab Inc.

San Francisco (CA)

On-site

USD 300,000 - 400,000

Full time

39 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
Relocation support

Job summary

Thinking Machines Lab Inc. is seeking a software engineer to build and scale the distributed systems powering Tinker, our post‑training platform.

You will design and operate systems that schedule training and sampling jobs across large GPU clusters, ensuring reliability and performance for real users. You will collaborate with research and ML infrastructure teams to evolve architecture for new workloads, drive incidents to resolution, and mentor others in distributed systems practices.

Qualifications

  • 7+ years of experience building, running, and scaling distributed systems in production, with on‑call ownership of live services.
  • Deep understanding of distributed systems fundamentals: consensus, consistency models, failure detection, scheduling, and state management under partial failure.
  • Track record of designing systems that operate reliably at scale, including diagnosing and resolving production incidents under time pressure.
  • Strong systems‑level coding ability in Python, Go, Rust, or C++, and comfort working close to the infrastructure layer.

Responsibilities

  • Design, build, and operate the distributed systems underlying Tinker, including job scheduling, resource orchestration, checkpointing, and fault recovery across large multi‑node GPU clusters.
  • Own the reliability, latency, and throughput of live production systems serving concurrent training and sampling workloads for external users.
  • Build for graceful degradation and fast recovery: handle node failures, preemptions, and network partitions without losing user progress or data.
  • Improve observability across the platform — metrics, tracing, and alerting that make it possible to detect and diagnose issues before or as they affect users.
  • Lead incident response for the systems you own, drive root‑cause analysis, and turn findings into lasting fixes.
  • Partner with research and ML infrastructure teams to understand new training and sampling workloads and to evolve the platform's architecture to support them at scale.
  • Make build‑vs‑buy and architectural tradeoff calls for core infrastructure components, and mentor other engineers on distributed systems practices.

Skills

Distributed systems
Python
Go
Rust
C++
Incident response
System design
Observability
Mentoring
On-call ownership

Tools

NCCL
RDMA

Job description

Thinking Machines Lab Inc. is seeking a software engineer to build and scale the distributed systems powering Tinker, our post‑training platform.

You will design and operate systems that schedule training and sampling jobs across large GPU clusters, ensuring reliability and performance for real users. You will collaborate with research and ML infrastructure teams to evolve architecture for new workloads, drive incidents to resolution, and mentor others in distributed systems practices.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

GPU Supercomputing Engineer for Large-Scale ML Training
GPU Supercomputing Engineer for Large-Scale ML Training

Bison • San Francisco (CA)

On-site
USD 350,000 - 475,000
Health/dental/vision
Unlimited PTO
Parental leave
+1
Distributed ML Training Infra Engineer
Distributed ML Training Infra Engineer

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Distributed AI Training Systems Engineer
Distributed AI Training Systems Engineer

Bison • San Francisco (CA)

On-site
USD 400,000 - 500,000
Health benefits
Unlimited PTO
Parental leave
+1
Software Engineer, Tinker Infrastructure
Software Engineer, Tinker Infrastructure

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Senior Distributed Systems Engineer for AI Clusters
Senior Distributed Systems Engineer for AI Clusters

Speedrun Talent Network • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health benefits
Dental benefits
Vision benefits
+3
Infrastructure Engineer - Scalable ML Systems
Infrastructure Engineer - Scalable ML Systems

Bison • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Training Infra — Unlimited PTO
Software Engineer, Training Infra — Unlimited PTO

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Parental leave
+1
Staff Engineer - Distributed AI Infrastructure & Systems
Staff Engineer - Distributed AI Infrastructure & Systems

Micro1 • United States

Remote
USD 180,000 - 260,000
Senior GPU Infrastructure Engineer: Schedulers & ML Infra
Senior GPU Infrastructure Engineer: Schedulers & ML Infra

Career Movement • San Francisco (CA)

On-site
USD 180,000 - 240,000
Site Reliability Engineer: AI Platform
Site Reliability Engineer: AI Platform

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1