AI Training Infrastructure Engineer – Unlimited PTO

Speedrun Talent Network

San Francisco (CA)

On-site

USD 300,000 - 400,000

Full time

43 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Dental insurance
Vision benefits
Unlimited PTO
Paid parental leave
Relocation support

Job summary

Thinking Machines is hiring a Software Engineer, Training Infrastructure to build the systems that train our frontier models and power Tinker, our fine‑tuning API. You will work on a distributed training stack that scales across large GPU clusters, handling scheduling, fault tolerance, and cost with reliability.

The role emphasizes optimizing throughput, memory usage, and kernel‑level tuning while supporting new open‑weights and in‑house models like mixture‑of‑experts.

Qualifications

  • 4+ years of software engineering experience in distributed systems.
  • Proficiency in Python and a systems language (C++, Rust, or Go).
  • Hands‑on experience with large-scale model training or inference using PyTorch.
  • Proven ability to debug complex performance and reliability issues in distributed environments.

Responsibilities

  • Design, build, and operate the distributed training infrastructure for model training.
  • Improve throughput and cost efficiency via profiling and optimization.
  • Support new models and architectures including mixture‑of‑experts.
  • Build reliability with automated failure detection, job recovery, and multi‑tenant isolation.
  • Collaborate with API, frameworks, and product teams to keep infrastructure cohesive.

Skills

Python
C++
Rust
Go
PyTorch
Distributed systems debugging

Tools

Kubernetes
Slurm

Job description

Thinking Machines is hiring a Software Engineer, Training Infrastructure to build the systems that train our frontier models and power Tinker, our fine‑tuning API. You will work on a distributed training stack that scales across large GPU clusters, handling scheduling, fault tolerance, and cost with reliability.

The role emphasizes optimizing throughput, memory usage, and kernel‑level tuning while supporting new open‑weights and in‑house models like mixture‑of‑experts.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer, Training Infra — Unlimited PTO
Software Engineer, Training Infra — Unlimited PTO

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Parental leave
+1
Distributed ML Training Infra Engineer
Distributed ML Training Infra Engineer

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Distributed AI Training Systems Engineer
Distributed AI Training Systems Engineer

Bison • San Francisco (CA)

On-site
USD 400,000 - 500,000
Health benefits
Unlimited PTO
Parental leave
+1
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Speedrun Talent Network • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Parental leave
+1
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
AI Training Infrastructure Engineer – Distributed GPU
AI Training Infrastructure Engineer – Distributed GPU

FDE Times • San Mateo (CA)

On-site
USD 140,000 - 230,000
Software Engineer, Tinker Infrastructure
Software Engineer, Tinker Infrastructure

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Infrastructure
Software Engineer, Infrastructure

Bison • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
AI Training Infrastructure Engineer
AI Training Infrastructure Engineer

OpenAI, Inc. • United States

Remote
USD 150,000 - 190,000