Senior Distributed Systems Engineer – Tinker Platform

Speedrun Talent Network

San Francisco (CA)

On-site

USD 300,000 - 400,000

Full time

36 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Dental insurance
Vision insurance
Unlimited PTO
Paid parental leave
Relocation support

Job summary

Thinking Machines is hiring a software engineer to build and scale distributed systems that power Tinker, our post-training platform. You will design and operate infrastructure for scheduling training and sampling jobs across large GPU clusters, ensuring fault tolerance, observability, and high availability for real users.

The role is hands-on with ownership, including architectural decisions, incident response, and setting direction for parts of the platform.

Qualifications

  • 7+ years of experience building, running, and scaling distributed systems in production.
  • Deep understanding of distributed systems fundamentals: consensus, consistency models, failure detection, scheduling, and state management under partial failure.
  • Track record of designing systems that operate reliably at scale and resolving production incidents under time pressure.
  • Strong systems-level coding ability in Python, Go, Rust, or C++, and comfort working close to the infrastructure layer.

Responsibilities

  • Design, build, and operate the distributed systems underlying Tinker, including job scheduling, resource orchestration, checkpointing, and fault recovery across large multi-node GPU clusters.
  • Own the reliability, latency, and throughput of live production systems serving concurrent training and sampling workloads for external users.
  • Build for graceful degradation and fast recovery: handle node failures, preemptions, and network partitions without losing user progress or data.
  • Improve observability across the platform—metrics, tracing, and alerting to detect and diagnose issues before or as they affect users.
  • Lead incident response for the systems you own, drive root-cause analysis, and turn findings into lasting fixes.
  • Partner with research and ML infrastructure teams to understand new training and sampling workloads and evolve the platform's architecture at scale.
  • Make build-vs-buy and architectural tradeoff calls for core infrastructure components, and mentor other engineers on distributed systems practices.

Skills

7+ years distributed systems
Distributed systems fundamentals
Production incident response
Languages: Python/Go/Rust/C++

Tools

Python
Go
Rust
C++

Job description

Thinking Machines is hiring a software engineer to build and scale distributed systems that power Tinker, our post-training platform. You will design and operate infrastructure for scheduling training and sampling jobs across large GPU clusters, ensuring fault tolerance, observability, and high availability for real users.

The role is hands-on with ownership, including architectural decisions, incident response, and setting direction for parts of the platform.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Distributed Systems Engineer, Scalable GPU Platform
Senior Distributed Systems Engineer, Scalable GPU Platform

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Tinker Infrastructure
Software Engineer, Tinker Infrastructure

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Tinker Infrastructure
Software Engineer, Tinker Infrastructure

Speedrun Talent Network • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision insurance
+3
Software Engineer, Training Infra — Unlimited PTO
Software Engineer, Training Infra — Unlimited PTO

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Parental leave
+1
Senior Distributed Systems Engineer for AI Clusters
Senior Distributed Systems Engineer for AI Clusters

Speedrun Talent Network • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health benefits
Dental benefits
Vision benefits
+3
Distributed ML Training Infra Engineer
Distributed ML Training Infra Engineer

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Site Reliability Engineer
Site Reliability Engineer

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health benefits
Dental benefits
Vision benefits
+3
Site Reliability Engineer: AI Platform
Site Reliability Engineer: AI Platform

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Site Reliability Engineer, Production
Site Reliability Engineer, Production

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Full-Stack AI Platform Engineer for Tinker
Full-Stack AI Platform Engineer for Tinker

Precision Labs • San Francisco (CA)

Hybrid
USD 300,000 - 475,000
Health benefits
Dental & Vision
Unlimited PTO
+2