SRE for AI Platform: Reliability at Scale

Mosaic.tech; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines is seeking a Site Reliability Engineer to drive the reliability of Tinker end-to-end. You will work alongside the platform and research teams to make every layer of the system more robust and resilient.

You will define and own end-to-end reliability, develop SLOs for distributed training systems, and design monitoring and incident response across the full training path, ensuring secure, multi-tenant operation at scale.

Qualifications

• Bachelor's degree or equivalent in CS/Engineering or related field.
• Experience in distributed systems, cloud infrastructure, or site reliability engineering.
• Proficiency writing software to solve reliability problems, including tooling and automation.
• Experience with production incident response, postmortems, and systematic reliability improvement.
• Strong communication and coordination across engineering and research teams.

Responsibilities

• Define and own end-to-end reliability, from CI/CD flows to production observability and incident response.
• Develop appropriate Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency with development velocity.
• Design and implement monitoring and observability across the full training path.
• Drive incident response for Tinker platform issues, ensuring rapid recovery and thorough incident reviews.
• Harden multi-tenant isolation and resource scheduling for efficient, reliable GPU workloads; collaborate with security teams.

Skills

Distributed systems

Cloud infrastructure

Software tooling & automation

Incident response

Cross-team collaboration

Education

Bachelor's degree or equivalent

Tools

Kubernetes

Cloud platforms

Job description

Thinking Machines is seeking a Site Reliability Engineer to drive the reliability of Tinker end-to-end. You will work alongside the platform and research teams to make every layer of the system more robust and resilient.

You will define and own end-to-end reliability, develop SLOs for distributed training systems, and design monitoring and incident response across the full training path, ensuring secure, multi-tenant operation at scale.