Lead GPU Fabric Observability Engineer

Baseten; New York (NY); On-site; USD 200,000 - 380,000; Full time

Benefits offered by this job

Competitive compensation and equity

Medical, dental, and vision coverage

Flexible PTO

Parental leave

401(k)

Learning opportunities

Job summary

Baseten is building its own GPU infrastructure for large-scale inference in the United States. As Lead Software Engineer, you will own the observability and root-cause analysis architecture for the GPU fabric, collecting signals from switches, hosts, and inference services to diagnose incidents in real time.

You will bridge networking and inference software, crafting topology models, service ownership maps, and proactive triage workflows to guide operators on remediation steps.

Qualifications

• Staff-level or senior staff-level experience building production infrastructure software.
• Strong distributed systems background, especially streaming systems, telemetry pipelines, diagnostics, or control-plane software.
• Experience building systems that process high-volume, high-cardinality, noisy operational data.
• Understanding of networking fundamentals and high-performance networks.
• Ability to work with low-level infrastructure signals and build practical correlation, anomaly detection, or root-cause analysis systems.

Responsibilities

• Own Baseten’s GPU fabric observability and root-cause analysis architecture.
• Build telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services.
• Model topology, flow paths, service ownership, and failure domains.
• Separate true fabric faults from host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload failures.

Skills

Distributed systems

Telemetry pipelines

Networking fundamentals

Observability

Root-cause analysis

Tools

Kubernetes

Job description

Baseten is building its own GPU infrastructure for large-scale inference in the United States. As Lead Software Engineer, you will own the observability and root-cause analysis architecture for the GPU fabric, collecting signals from switches, hosts, and inference services to diagnose incidents in real time.

You will bridge networking and inference software, crafting topology models, service ownership maps, and proactive triage workflows to guide operators on remediation steps.