Post-Training AI SRE: Keep RL Pipelines Fast & Reliable

Bison

San Francisco (CA)

On-site

USD 300,000 - 350,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health benefits
Unlimited PTO
Parental leave
Relocation support

Job summary

Thinking Machines in San Francisco, CA, is hiring a Site Reliability Engineer to own the health of post-training and RL systems. You will collaborate with researchers, debug complex failures, and build automation to keep runs fast and reliable.

The role demands 4+ years in production engineering and strong Python/Go/C++ skills, with a focus on distributed systems. You’ll operate GPUs/TPU clusters, improve fault tolerance, and participate in on-call rotations.

Qualifications

  • 4+ years as a production or site reliability engineer on large-scale systems.
  • Ability to debug complex distributed failures across networks, hardware, and schedulers.
  • Proficient in Python and Go/C++, with pragmatic scripting as needed.
  • Strong grounding in Linux internals and networking fundamentals.
  • Experience owning production systems, including on-call rotations.

Responsibilities

  • Own the reliability, performance, and uptime of large-scale post-training RL jobs from launch to completion.
  • Partner with research teams during active model runs to unblock training and speed iteration.
  • Debug failures across the full stack and drive root-cause analysis.
  • Build monitoring, alerting, and self-healing automation to reduce stalls.
  • Improve checkpointing and fault tolerance; optimize job scheduling and resource use.

Skills

Python
Go/C++
Linux internals
On-call rotations
Distributed systems
Diagnostics and debugging

Tools

Slurm
Kubernetes
PyTorch
Ray
NCCL
InfiniBand

Job description

Thinking Machines in San Francisco, CA, is hiring a Site Reliability Engineer to own the health of post-training and RL systems. You will collaborate with researchers, debug complex failures, and build automation to keep runs fast and reliable.

The role demands 4+ years in production engineering and strong Python/Go/C++ skills, with a focus on distributed systems. You’ll operate GPUs/TPU clusters, improve fault tolerance, and participate in on-call rotations.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Site Reliability Engineer, Post Training
Site Reliability Engineer, Post Training

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health benefits
Unlimited PTO
Parental leave
+1
Site Reliability Engineer: AI Platform
Site Reliability Engineer: AI Platform

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Staff ML Engineer - Post-Training & RL Pipelines
Staff ML Engineer - Post-Training & RL Pipelines

SF Tensor • San Francisco (CA)

On-site
USD 275,000 - 315,000
SRE - AI Platform Reliability & Observability
SRE - AI Platform Reliability & Observability

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health benefits
Dental benefits
Vision benefits
+3
ML Systems Engineer — RL & Post-Training Pipelines
ML Systems Engineer — RL & Post-Training Pipelines

Remanence • Paris (TX)

Hybrid
USD 135,000 - 203,000
Visa sponsorship
Relocation support
Hybrid setup
Embedded R&D Software Engineer for Post-Training AI
Embedded R&D Software Engineer for Post-Training AI

Bison • San Francisco (CA)

On-site
USD 350,000 - 475,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Senior SRE: AI Training & Inference Infra
Senior SRE: AI Training & Inference Infra

ByteDance • San Jose (CA)

On-site
USD 254,000 - 480,000
Medical, dental, and vision insurance
401(k) with company match
Paid parental leave
+3
SRE: Scalable ML Infra & CI/CD Architect
SRE: Scalable ML Infra & CI/CD Architect

Baseten • San Francisco (CA)

On-site
USD 165,000 - 330,000
Competitive compensation with equity
100% medical, dental, and vision coverage
Generous PTO including Winter Break
+2
Sr. Site Reliability Engineer
Sr. Site Reliability Engineer

Tiger Analytics • Vermont

Hybrid
USD 100,000 - 140,000
Remote Senior SRE: Build Reliable, Scalable AI Infra
Remote Senior SRE: Build Reliable, Scalable AI Infra

Runware • Town of Sweden (NY)

Remote
USD 140,000 - 190,000
Generous paid time off
Meaningful stock options
Remote-first setup
+3