Senior ML Infrastructure Engineer - High-Throughput Inference

SambaNova Systems; San Jose (CA); On-site; USD 180,000 - 250,000; Full time

Benefits offered by this job

Flexible PTO in US

Parental Leave

Benefits (medical, dental and vision)

Flexible Spending Accounts

401k/Pensions

Gym Access

Flexible Working Hours

Job summary

SambaNova Systems in California is seeking a Senior Software Engineer, ML Infrastructure to design, build, and operate production-grade inference infrastructure powering the SambaNova serving stack on the Reconfigurable Dataflow Unit (RDU).

You will own end-to-end systems from request scheduling to accuracy infrastructure, collaborating with ML, compiler, runtime, and product teams to ship inference features from prototype to production.

Qualifications

• Experience designing and maintaining production services with strict latency, throughput, and availability requirements.
• Working knowledge of modern LLM inference techniques and familiarity with open-source serving stacks such as vLLM, TensorRT-LLM, or SGLang.
• Experience collaborating across teams to deliver complex, system-level engineering solutions.
• 5+ years of industry experience building and operating large-scale distributed systems, ideally in ML serving.
• B.S. in Computer Science, Electrical Engineering, or related field.
• Strong software engineering fundamentals: algorithms, data structures, concurrency, and systems design.
• Proficiency in Python.

Responsibilities

• Design and operate production-grade inference infrastructure powering SambaNova’s serving stack on RDU.
• Own end-to-end systems from request scheduling to accuracy infrastructure.
• Collaborate with ML, compiler, runtime, and product teams to ship inference features.
• Integrate SambaNova with vLLM and adjacent serving frameworks.
• Contribute to technical design discussions and code reviews.

Skills

LLM inference

Python

Distributed systems

Algorithms

Data structures

Concurrency

Systems design

Cross-team collaboration

Education

B.S. in CS/EE or related field

Tools

vLLM

TensorRT-LLM

SGLang

Job description

SambaNova Systems in California is seeking a Senior Software Engineer, ML Infrastructure to design, build, and operate production-grade inference infrastructure powering the SambaNova serving stack on the Reconfigurable Dataflow Unit (RDU).

You will own end-to-end systems from request scheduling to accuracy infrastructure, collaborating with ML, compiler, runtime, and product teams to ship inference features from prototype to production.