Software Engineer, ML Infra

Thinking Machines Lab; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health benefits

Dental benefits

Vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines seeks a Software Engineer to bridge research and infrastructure, offering a generalist role with impact on ML compute and hero-run decisions. You’ll contribute across kernel, networking, and application layers, mentoring teammates and shaping scalable, reliable systems for frontier AI workloads.

The role emphasizes hands-on incident support, postmortems, and cross-team collaboration, with a strong focus on operational excellence and rapid, correct judgment under pressure.

Qualifications

• Credible, hands-on competence in 4 or more of the following areas: Linux kernel, networking, GPUs/CUDA, distributed systems runtimes, storage, compilers/language runtimes, observability internals.
• Comfort operating without a clearly defined owner, and the judgment to know when to dig in yourself versus escalation.

Responsibilities

• Debug across the full stack — kernel, NCCL, scheduler, application, and telemetry — often in the same afternoon, to find root causes that don't show up in any single layer
• Provide embedded, hands-on support during hero runs and major incidents, staying with a problem until it's genuinely resolved
• Serve as the front door for researchers when something's broken and it isn't obvious who owns it
• Lead postmortems and build the tooling that prevents the next incident — acting as a force multiplier, not just a responder
• Mentor other engineers into this kind of cross-stack breadth

Skills

Linux kernel

Networking

GPUs / CUDA

Distributed systems runtimes

Storage

Compilers / language runtimes

Observability internals

Job description

About Thinking Machines

The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.

About the Role

We're hiring a Software Engineer to sit at the day-to-day interface between research and infrastructure. This is a generalist role with broad scope: you'll be one of the people always in the room for infra decisions on the ML systems side, with a clear enough view of upcoming compute needs to see support burden coming before it arrives.

You'll also be one of the people who babysits hero runs — the ones at 2am when a 4k-GPU job hits a weird Xid and someone needs to decide, quickly and correctly, whether to drain the node, restart the job, or escape to NVIDIA before the run loses checkpoints. That kind of judgment, built across the kernel, the network, the scheduler, and the application layer, is the core of the job.

What You'll Do

• Debug across the full stack — kernel, NCCL, scheduler, application, and telemetry — often in the same afternoon, to find root causes that don't show up in any single layer

• Provide embedded, hands-on support during hero runs and major incidents, staying with a problem until it's genuinely resolved

• Serve as the front door for researchers when something's broken and it isn't obvious who owns it

• Lead postmortems and build the tooling that prevents the next incident — acting as a force multiplier, not just a responder

• Mentor other engineers into this kind of cross-stack breadth

Skills & Qualifications

Minimum Qualifications

• Credible, hands-on competence in 4 or more of the following: Linux kernel, networking, GPUs / CUDA, distributed systems runtimes, storage, compilers / language runtimes, observability internals — at this level, this required range is the qualifying signal

• Comfort operating without a clearly defined owner, and the judgment to know when to dig in yourself versus when to
  

Preferred Qualifications

• Track record of being the person other engineers …

• Shipped meaningful contributions in 3+ distinct technical stacks

• Track record of leading major incidents where the root cause was non-obvious

• Researcher-facing comfort: can talk to a researcher about their workload without making them feel dumb, and can tell them no when the right answer is no

• Experience operating at the scale of frontier training or inference clusters, and appetite for owning the hardest, least-defined problems in that stack

Logistics

• Location: This role is based in San Francisco, CA.
• Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000 – $475,000 USD, plus equity.
• Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
• Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.