Lead GPU Fabric Observability Engineer

Baseten; San Francisco (CA); On-site; USD 200,000 - 380,000; Full time

Benefits offered by this job

Competitive compensation

Meaningful equity

Medical, dental, and vision insurance

Flexible PTO

Parental leave

401(k)

Exposure to ML startups

Job summary

Baseten is seeking a Lead Software Engineer to own GPU fabric observability and RCA at scale in San Francisco. You will drive real-time telemetry, cross-layer diagnosis, and actionable guidance for operators during incidents.

The role sits at the intersection of networking and inference software, requiring deep systems expertise, practical correlation, and a passion for solving hard distributed problems with measurable impact.

Qualifications

• Staff-level or senior production infra experience.
• Strong background in distributed systems & telemetry.
• Experience handling high-volume operational data.
• Familiarity with networking fundamentals.
• Ability to diagnose issues across fabric, host, NIC, GPU, or RDMA layers.

Responsibilities

• Own GPU fabric observability and RCA architecture.
• Build telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services.
• Model topology, flow paths, service ownership, and failure domains.
• Separate true fabric faults from host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload failures.
• Create operator workflows for triage, remediation, and post-incident learning.

Skills

Distributed systems

Telemetry pipelines

Diagnostics

Networking basics

Root-cause analysis

Tools

Kubernetes

Job description

Baseten is seeking a Lead Software Engineer to own GPU fabric observability and RCA at scale in San Francisco. You will drive real-time telemetry, cross-layer diagnosis, and actionable guidance for operators during incidents.

The role sits at the intersection of networking and inference software, requiring deep systems expertise, practical correlation, and a passion for solving hard distributed problems with measurable impact.