GPU Networking Engineer for Large-Scale AI Fabric

Thinking Machines Lab Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. is seeking a network engineer to own the lowest layers of the network stack for large-scale training and inference. You will ensure interconnect reliability across GPU fabrics, debugging NICs, and building instrumentation for faster troubleshooting.

The role is hands-on and cross-stack, working with cloud providers to drive issues to resolution and to keep researchers’ fleets trusted and fast.

Qualifications

• Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
• Proficiency in at least one backend language (we use Python or Rust).
• Experience operating large-scale clusters and container orchestration systems (e.g. Kubernetes or Slurm).
• Comfort operating across the stack and owning projects end-to-end.
• Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
• A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

Responsibilities

• Reason about and validate GPU network fabric design across our deployments.
• Debug RDMA / RoCEv2 across different NIC vendors. Diagnose collective failures of production NCCL, PFC/ECN tuning, and congestion control behavior.
• Own NVLink / NVSwitch interconnect - including fabric manager and IMEX health, link and lane errors, and how the GPU fabric interacts with collectives.
• Build host-level network instrumentation and use Linux tooling to build dashboards and alerts, not just the bug report.
• Navigate cross-cloud fabric quirks across providers and triage across the NIC, driver, kernel, switch, and workload boundaries.
• Drive escalations with cloud-provider networking teams, owning issues end-to-end until they're resolved.

Skills

Distributed systems

Backend development

Linux fundamentals

Cross-functional collaboration

Education

Bachelor's degree in CS/Engineering

Tools

Python

Rust

Kubernetes

Slurm

NCCL

Job description

Thinking Machines Lab Inc. is seeking a network engineer to own the lowest layers of the network stack for large-scale training and inference. You will ensure interconnect reliability across GPU fabrics, debugging NICs, and building instrumentation for faster troubleshooting.

The role is hands-on and cross-stack, working with cloud providers to drive issues to resolution and to keep researchers’ fleets trusted and fast.