Reliability Engineer, Supercomputing

Mosaic.tech; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines is hiring an engineer to ensure the reliability of our GPU supercomputing fleet, owning the seam between hardware, firmware, and operating system. You will diagnose hardware anomalies, track root causes to the hardware, and coordinate fixes with vendors so researchers can run at scale.

Based in San Francisco, this full-time role requires owning drivers, kernel surfaces, and diagnostics, plus automating fleet monitoring and reliability improvements across multi-disciplinary

Qualifications

• Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
• Proficiency in Python or Rust and ability to own projects end-to-end.
• Experience operating large-scale clusters and container orchestration (Kubernetes/Slurm).
• Comfortable owning end-to-end projects and working across different stacks and teams.
• Fluency in debugging hardware issues and collaborating with hardware vendors.

Responsibilities

• Investigate, reproduce, and remediate issues across large GPU clusters.
• Own the drivers, kernel surface, and diagnostics that span hardware, firmware, and OS.
• Automate the monitoring of fleet reliability and analyze error rates to validate fixes.
• Drive the firmware lifecycle: tracking, qualification, staged rollout, and regression analysis.
• Engage vendors directly to get real fixes; manage RMA flows when hardware needs replacement.
• Monitor and improve GPU hardware health signals and turn them into actionable improvements.
• Write postmortems and vendor cases to move issues forward.

Skills

Python

Rust

Kubernetes

Slurm

Linux systems

Hardware debugging

Education

Bachelor’s degree or equivalent experience

Tools

Kubernetes

Slurm

BMC / iDRAC / IPMI / Redfish

Job description

About Thinking Machines

The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.

About the Role

We're hiring an engineer to ensure the reliability of our GPU supercomputing fleet, owning the seam between hardware, firmware, and operating system. You will track the long tail of hardware issues: We are conducting frontier research in AI and a single bad NIC, HBM or a kernel driver edge case can compromise an experiment. Your job is to diagnose these issues, track their root cause down to the hardware, and resolve them internally or directly with vendors so that our researchers can run at scale and with confidence.

What You’ll Do

• Investigate, reproduce, and remediate issues across large GPU clusters.

• Own the drivers, kernel surface, and diagnostics that span hardware, firmware, and OS.

• Automate the monitoring of fleet reliability and analyze error rates to validate whether a fix or firmware change measurably reduced failures rather than shifting them around.

• Drive the firmware lifecycle: tracking, qualification, staged rollout, and regression analysis.

• Engage vendors directly — GPUs, server OEMs, NIC vendors, and storage vendors — to get real fixes rather than ticket numbers. Manage RMA flows when hardware needs to come out.

• Monitor and improve GPU hardware health signals and turn them into actionable reliability improvements.

• Write clear postmortems and vendor cases that move issues forward.

Skills and Qualifications

Minimum qualifications:

• Bachelor’s degree or equivalent experience in computer science, engineering, or similar.

• Proficiency in at least one backend language (we use Python or Rust).

• Experience operating large-scale clusters and container orchestration systems (e.g. Kubernetes or Slurm).

• Comfort operating across the stack and owning projects end-to-end.

• Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.

• A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

Preferred qualifications — we encourage you to apply if you meet some but not all of these:

• Fluency with Linux systems and debugging tools.

• Proven statistical rigor in analyzing reliability.

• A track record of debugging a problem from application symptom to the root cause in hardware.

• Comfort reading vendor errata, firmware release notes, and kernel changelogs.

• Experience engaging hardware vendors directly — not just through escalation portals.

• Linux kernel literacy: the scheduler, memory management, IRQ paths, and the driver model.

• Out-of-band management experience: BMC / iDRAC / IPMI / Redfish.

• Depth in GPU hardware health: Xid error taxonomy, NVLink, NVSwitch, fabric manager, and DCGM.

• Proficiency in at least one backend language (we use Python and Rust).

• Significant ownership of the hardware reliability function at scale.

• Strong writing skills for vendor cases and postmortems.

• An instinct for telling apart a flaky machine, a flaky workload, and a flaky test.

Logistics

• Location: This role is based in San Francisco, California.

• Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 - $475,000 USD.

• Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.

• Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.