GPU Networking Engineer — RDMA & Distributed Inference

Baseten; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Baseten is building the global operating system for distributed AI workloads. We are seeking foundational engineers to lead our GPU networking efforts, integrating RDMA into the inference stack and enabling scalable, low-latency distributed serving of large models.

You will architect software fabric unifying thousands of GPUs, tapping into Open-Ecosystem while building from scratch when needed to unlock next-gen performance and startup speeds.

Responsibilities

• Make RDMA First-Class: You will work on integrating RDMA/RoCE/InfiniBand capabilities directly into our inference stack, helping us move beyond TCP/IP to unlock order-of-magnitude improvements in bandwidth and latency.
• Optimize Distributed Inference: You will implement and tune the networking layers necessary for efficient Disaggregated KV Cache Offload and WideEP, ensuring seamless communication across NVLink and InfiniBand for our MoE models.
• Enable Serverless-Grade Startup Speeds for LLMs: You will work deeply with checkpointing and storage mechanisms to enable sub-10-second startup for trillion-parameter models.
• Deep-Dive into Hardware: You will characterize and validate networking performance on bleeding-edge clusters (H100/H200, B200/B300, GB200/300 NVL72), writing the acceptance tests that ensure our hardware delivers peak achievable throughput and minimal latency.
• Build Observability: You will design the tools that let us visualize packet flow, congestion, and effective bandwidth across the GPU interconnects, helping us diagnose complex distributed system behaviors.

Skills

RDMA

InfiniBand

NVLink

Distributed systems

Networking stack

C/C++

Performance optimization

Job description

Baseten is building the global operating system for distributed AI workloads. We are seeking foundational engineers to lead our GPU networking efforts, integrating RDMA into the inference stack and enabling scalable, low-latency distributed serving of large models.

You will architect software fabric unifying thousands of GPUs, tapping into Open-Ecosystem while building from scratch when needed to unlock next-gen performance and startup speeds.