SRE for Scalable AI Platform Reliability & Incident Response

Thinking Machines Lab; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health/Dental/Vision

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines Lab is seeking a Site Reliability Engineer (SRE) to drive the reliability of Tinker end-to-end, collaborating with platform engineers and researchers to make every layer of the system robust.

The role emphasizes defining end-to-end reliability, building observability, incident response, and multi-tenant isolation for large-scale distributed training workloads on Kubernetes.

Qualifications

• Bachelor's degree or equivalent experience in computer science, engineering, or similar.
• Experience in distributed systems, cloud infrastructure, or site reliability engineering.
• Proficiency writing software to solve reliability problems, including building tooling and automation.
• Experience with production incident response, postmortems, and systematic reliability improvement.
• Strong communication skills and track record of coordination across engineering and research teams.

Responsibilities

• Define and own end-to-end reliability, from CI/CD flows to production observability and incident response.
• Develop appropriate Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency with development velocity.
• Design and implement monitoring and observability across the full training path.
• Drive incident response for Tinker platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence.
• Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation.
• Collaborate with security teams to address production vulnerabilities.

Skills

Distributed systems

Cloud infrastructure

Software tooling

Incident response

Cross-team communication

Education

Bachelor's degree or equivalent in CS/Engineering

Tools

Kubernetes

CI/CD tooling

Observability tooling

Job description

Thinking Machines Lab is seeking a Site Reliability Engineer (SRE) to drive the reliability of Tinker end-to-end, collaborating with platform engineers and researchers to make every layer of the system robust.

The role emphasizes defining end-to-end reliability, building observability, incident response, and multi-tenant isolation for large-scale distributed training workloads on Kubernetes.