ML Infrastructure Engineer - Fast, Reliable RL Pipelines

Thinking Machines Lab; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines is hiring an Infrastructure Engineer to keep post-training and RL systems fast, reliable, and easy for researchers to iterate on. You’ll own the health of the training runs, clusters, and pipelines powering post-training and RL at Thinking Machines.

You’ll work with research teams during active model runs, debug failures in real time, and build tooling that lets researchers focus on science rather than babysitting jobs.

Qualifications

• 4+ years of experience as a production engineer, site reliability engineer, or infrastructure engineer operating large-scale distributed systems in production.
• Track record debugging complex failures across distributed systems — networking, hardware, kernel, or scheduler issues.
• Strong software engineering skills in Python and/or Go/C++, with the judgment to know when to script a fix versus build a system.
• Solid grounding in Linux systems internals and networking fundamentals.
• Comfortable owning production systems, including participating in on-call rotations.

Responsibilities

• Own the reliability, performance, and uptime of large-scale post-training and RL training jobs, from launch through completion.
• Partner directly with research teams during active model runs, embedding with them to unblock training and speed up iteration.
• Debug failures across the full stack — accelerators, networking, storage, schedulers, and training frameworks — and drive issues to root cause.
• Build monitoring, alerting, and automated recovery so runs self-heal or fail fast instead of silently stalling.
• Improve checkpointing, fault tolerance, and job scheduling so hardware failures cost minutes, not days of compute.
• Build internal tools that reduce toil and improve cluster utilization across post-training and RL workloads.
• Participate in an on-call rotation supporting production model runs.
• Write postmortems and turn recurring failure patterns into permanent infrastructure fixes

Skills

Production engineering

Distributed systems debugging

Python

Go/C++

Linux networking

On-call ownership

Tools

PyTorch

Ray

Slurm

Kubernetes

Job description

Thinking Machines is hiring an Infrastructure Engineer to keep post-training and RL systems fast, reliable, and easy for researchers to iterate on. You’ll own the health of the training runs, clusters, and pipelines powering post-training and RL at Thinking Machines.

You’ll work with research teams during active model runs, debug failures in real time, and build tooling that lets researchers focus on science rather than babysitting jobs.