Site Reliability Engineer

Thinking Machines Lab

San Francisco (CA)

On-site

USD 300,000 - 400,000

Full time

1 hour ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health benefits
Dental benefits
Vision benefits
Unlimited PTO
Paid parental leave
Relocation support

Job summary

Thinking Machines Lab is seeking a Site Reliability Engineer to drive the reliability of Tinker end-to-end. You will work with platform engineers and researchers to strengthen every layer of the system, enabling robust, scalable, and secure AI training workflows.

The ideal candidate has strong SRE experience with distributed systems, cloud infrastructure, and automation tooling, and excels at cross-team collaboration to reduce incidents and improve performance.

Qualifications

  • Bachelor's degree in computer science, engineering, or related field or equivalent experience.
  • Experience with distributed systems, cloud infrastructure, and SRE practices.
  • Proficiency in building tooling and automation to improve reliability.
  • Experience with production incident response, postmortems, and improvements.
  • Strong communication and cross-team coordination skills.

Responsibilities

  • Define and own end-to-end reliability, from CI/CD flows to production observability and incident response.
  • Develop Service Level Objectives for distributed training systems balancing reliability and latency with velocity.
  • Design and implement monitoring and observability across the full training path.
  • Lead incident response for platform issues ensuring rapid recovery and thorough postmortems.
  • Harden multi-tenant isolation and resource scheduling for efficient, reliable workloads.
  • Collaborate with security teams to address production vulnerabilities.

Skills

Distributed systems
Cloud infrastructure
Site reliability engineering
Automation tooling
Incident response
Communication

Education

Bachelor's degree or equivalent

Tools

Kubernetes
CI/CD tooling

Job description

We're looking for a Site Reliability Engineer (SRE) to drive the reliability of Tinker end-to-end. You'll work alongside the engineers building the platform and research teams to make every layer of the system more robust and resilient.

About Tinker

Tinker is our fine-tuning API that empowers researchers and developers to customize frontier AI to their needs — opening access to capabilities that have previously been concentrated in a handful of labs. We manage the infrastructure while allowing Tinkerers full flexibility in training open weights models with their own data, algorithms, and for their own needs. Tinker is rapidly adding new customers, features, and novel use-cases. We’re hiring to grow the platform alongside the Tinker community.

What You’ll Do
  • Define and own end-to-end reliability, from CI/CD flows to production observability and incident response.
  • Develop appropriate Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency with development velocity.
  • Design and implement monitoring and observability across the full training path.
  • Drive incident response for Tinker platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence.
  • Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation
  • Collaborate with security teams to address production vulnerabilities
Skills and Qualifications
  • Bachelor's degree or equivalent experience in computer science, engineering, or similar.
  • Experience in distributed systems, cloud infrastructure, or site reliability engineering.
  • Proficiency writing software to solve reliability problems, including building tooling and automation.
  • Experience with production incident response, postmortems, and systematic reliability improvement.
  • Strong communication skills and track record of coordination across engineering and research teams.
Preferred qualifications
  • Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services)
  • Background in distributed training frameworks and how infrastructure failures surface in training behavior.
  • Track record building checkpoint and recovery systems for long-running distributed jobs.
  • Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads.
Logistics
  • Location: This role is based in San Francisco, California.
  • Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $300,000- $400,000 USD.
  • Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
  • Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Site Reliability Engineer, Production
Site Reliability Engineer, Production

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Training Infrastructure
Software Engineer, Training Infrastructure

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Software Engineer, Tinker Product
Software Engineer, Tinker Product

Precision Labs • San Francisco (CA)

Hybrid
USD 300,000 - 475,000
Health benefits
Dental & Vision
Unlimited PTO
+2
Site Reliability Engineer: AI Platform
Site Reliability Engineer: AI Platform

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
SRE - AI Platform Reliability & Observability
SRE - AI Platform Reliability & Observability

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health benefits
Dental benefits
Vision benefits
+3
Site Reliability Engineer, Post Training
Site Reliability Engineer, Post Training

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health benefits
Unlimited PTO
Parental leave
+1
Research, Tinker, RL Systems
Research, Tinker, RL Systems

Bison • San Francisco (CA)

On-site
USD 350,000 - 475,000
Health benefits
Dental benefits
Vision benefits
+3
Software Engineer, Infrastructure
Software Engineer, Infrastructure

Bison • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health, dental, and vision benefits
Unlimited PTO
Paid parental leave
+1
Software Engineer, Systems Generalist
Software Engineer, Systems Generalist

Bison • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health benefits
Dental benefits
Vision benefits
+3
Software Engineer, Research Tools
Software Engineer, Research Tools

Bison • New York (NY)

On-site
USD 300,000 - 350,000
Health insurance
Dental insurance
Vision coverage
+3