Lead Software Engineer - GPU Fabric Telemetry & Diagnostics

BaseTen; New York, San Francisco (NY, CA); On-site; USD 160,000 - 260,000; Full time

Benefits offered by this job

Competitive compensation

100% health insurance coverage

Flexible PTO including Winter Break

Parental leave

Fertility and family-building stipend

401(k)

Exposure to ML startups

Job summary

Baseten is seeking a Lead Software Engineer to design and build an observability and root-cause analysis system for GPU fabrics. You will integrate telemetry from switches, hosts, GPUs, and services, enabling real-time diagnosis as incidents unfold.

The role sits at the intersection of networking and inference software, tackling high-density NVIDIA systems and challenging failures with evidence-driven guidance for operators.

Qualifications

• Staff-level or senior staff-level experience building production infrastructure software.
• Strong distributed systems background, especially streaming systems, telemetry pipelines, diagnostics, or control-plane software.
• Experience building systems that process high-volume, high-cardinality, noisy operational data.
• Understanding of networking fundamentals and high-performance networks.
• Ability to work with low-level infrastructure signals and build practical correlation, anomaly detection, or root-cause analysis systems.

Responsibilities

• Own Baseten’s GPU fabric observability and root-cause analysis architecture.
• Build telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services.
• Model topology, flow paths, service ownership, and failure domains.
• Separate true fabric faults from host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload failures.
• Create clear operator workflows for triage, remediation, and post-incident learning.

Skills

Distributed systems

Telemetry & observability

Root-cause analysis

Networking fundamentals

Low-level instrumentation

Job description

Baseten is seeking a Lead Software Engineer to design and build an observability and root-cause analysis system for GPU fabrics. You will integrate telemetry from switches, hosts, GPUs, and services, enabling real-time diagnosis as incidents unfold.

The role sits at the intersection of networking and inference software, tackling high-density NVIDIA systems and challenging failures with evidence-driven guidance for operators.