SRE for AI Training Pipelines & RL Runs

Mosaic.tech; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Visa sponsorship

Relocation support

Unlimited PTO

Paid parental leave

Job summary

Thinking Machines is hiring a Site Reliability Engineer in San Francisco to own the health of post-training and RL pipelines, partnering with researchers to unblock training and harden infrastructure. Youll implement recovery, observability, and tooling to keep runs fast and reliable.

Youll operate in a high-scale ML environment, collaborate across teams, and participate in on-call rotations while driving permanent fixes and reducing toil for researchers.

Qualifications

• 4+ years in production engineering or SRE for large-scale distributed systems.
• Proven debugging across distributed systems including networks and schedulers.
• Strong Python and/or Go/C++ programming skills with practical fixes.
• Solid Linux internals and networking fundamentals.
• Comfortable owning production systems and on-call rotations.

Responsibilities

• Own reliability, performance, and uptime of large-scale post-training and RL jobs.
• Collaborate with researchers during active model runs to unblock training.
• Debug failures across accelerators, networking, storage, and schedulers.
• Build monitoring, alerting, and automated recovery for self-healing runs.
• Improve checkpointing, fault tolerance, and job scheduling efficiency.
• Develop internal tools to reduce toil and improve cluster utilization.
• Participate in on-call rotation and write postmortems for fixes.

Skills

Production engineering

SRE / site reliability

Python

Go/C++

Linux networking

On-call ownership

Debugging distributed systems

Tools

PyTorch

Ray

Kubernetes

Slurm

NCCL

InfiniBand

Job description

Thinking Machines is hiring a Site Reliability Engineer in San Francisco to own the health of post-training and RL pipelines, partnering with researchers to unblock training and harden infrastructure. Youll implement recovery, observability, and tooling to keep runs fast and reliable.

Youll operate in a high-scale ML environment, collaborate across teams, and participate in on-call rotations while driving permanent fixes and reducing toil for researchers.