Distributed ML Training Infra Engineer

Thinking Machines Lab

San Francisco (CA)

On-site

USD 300,000 - 400,000

Full time

4 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Dental insurance
Vision benefits
Unlimited PTO
Parental leave
Relocation support

Job summary

Thinking Machines is hiring a Software Engineer, Training Infrastructure to build the systems that train our frontier models and power Tinker, our fine-tuning API. You will work on the distributed training stack running supervised and reinforcement learning jobs across large GPU clusters.

You will ensure scale, reliability, and cost-efficiency while enabling researchers and developers to run complex training loops with strong abstractions and tooling.

Qualifications

  • 4+ years of software engineering experience in production distributed systems.
  • Strong programming skills in Python and systems languages such as C++, Rust, or Go.
  • Hands-on experience with large-scale model training or inference infrastructure (e.g., PyTorch).
  • Track record of debugging performance and reliability issues in distributed environments.

Responsibilities

  • Design, build, and operate the distributed training infrastructure for Tinker and frontier model training.
  • Improve training throughput and cost efficiency via profiling and memory optimization.
  • Bring up training support for new model architectures including mixture-of-experts.
  • Ensure reliability with automated failure detection, job recovery, and multi-tenant isolation.
  • Support multi-tenant LoRA fine-tuning and RL workloads; collaborate with API and product teams.

Skills

Python
C++
Rust
Go
PyTorch
Distributed systems

Tools

Kubernetes
Slurm

Job description

Thinking Machines is hiring a Software Engineer, Training Infrastructure to build the systems that train our frontier models and power Tinker, our fine-tuning API. You will work on the distributed training stack running supervised and reinforcement learning jobs across large GPU clusters.

You will ensure scale, reliability, and cost-efficiency while enabling researchers and developers to run complex training loops with strong abstractions and tooling.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer, Training Infra — Unlimited PTO
Software Engineer, Training Infra — Unlimited PTO

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Parental leave
+1
Distributed AI Training Systems Engineer
Distributed AI Training Systems Engineer

Bison • San Francisco (CA)

On-site
USD 400,000 - 500,000
Health benefits
Unlimited PTO
Parental leave
+1
AI Training Infrastructure Engineer – Unlimited PTO
AI Training Infrastructure Engineer – Unlimited PTO

Speedrun Talent Network • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Infrastructure Engineer - Scalable ML Systems
Infrastructure Engineer - Scalable ML Systems

Bison • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Speedrun Talent Network • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Parental leave
+1
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Senior Distributed Systems Engineer, Scalable GPU Platform
Senior Distributed Systems Engineer, Scalable GPU Platform

Thinking Machines Lab Inc. • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Distributed ML Infra Engineer for Large-Scale Pre-Training
Distributed ML Infra Engineer for Large-Scale Pre-Training

Mohamed bin Zayed University of Artificial Intelligence • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, vision
Bonus
401K Plan
+4
Distributed ML Infra Engineer - Large-Scale Pre-Training
Distributed ML Infra Engineer - Large-Scale Pre-Training

Institute of Foundation Models • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, and vision benefits
Bonus
401K Plan
+4