Senior ML Infra Engineer: High-Throughput AI Inference

SambaNovaSystems; United States; On-site; USD 200,000 - 275,000; Full time

Benefits offered by this job

Health insurance

Health Savings Account (HSA)

Headspace subscription

Gympass+ membership

One Medical membership

Job summary

SambaNova Systems is seeking a Senior Software Engineer, ML Infrastructure to design, build, and operate the production-grade inference infrastructure powering SambaNova's serving stack on the Reconfigurable Dataflow Unit (RDU) architecture. You will own end-to-end systems from request scheduling to accuracy gating, delivering reliable, high-throughput, low-latency services for SambaStack and SambaCloud.

This senior IC collaborates with ML, compiler, runtime, and product teams to ship inference

Qualifications

• Bachelor's degree in Computer Science, Electrical Engineering, or related field.
• 5+ years of industry experience building and operating large-scale distributed systems.
• Experience with latency, throughput, and availability requirements.

Responsibilities

• Design and productionize advanced inference techniques on RDU for performance and cost.
• Own SambaNova's integration with vLLM and adjacent serving frameworks for RDU.
• Own the public inference API surface exposed through SambaStack and SambaCloud.
• Build and maintain the accuracy verification and regression infrastructure gated for customers.
• Partner with ML, compiler, runtime, and product teams to take features from prototype to production.
• Contribute to technical design discussions and architectural decisions as a senior contributor.

Skills

Distributed systems

Python

LLM inference

System design

Collaboration

Open-source stacks

Education

Bachelor's degree in CS/EE or related field

Tools

vLLM

TensorRT-LLM

SGLang

Job description

SambaNova Systems is seeking a Senior Software Engineer, ML Infrastructure to design, build, and operate the production-grade inference infrastructure powering SambaNova's serving stack on the Reconfigurable Dataflow Unit (RDU) architecture. You will own end-to-end systems from request scheduling to accuracy gating, delivering reliable, high-throughput, low-latency services for SambaStack and SambaCloud.

This senior IC collaborates with ML, compiler, runtime, and product teams to ship inference