Senior ML Infrastructure Engineer

SambaNova Systems; San Jose (CA); On-site; USD 180,000 - 250,000; Full time

Benefits offered by this job

Flexible PTO in US

Parental Leave

Benefits (medical, dental and vision)

Flexible Spending Accounts

401k/Pensions

Gym Access

Flexible Working Hours

Job summary

SambaNova Systems in California is seeking a Senior Software Engineer, ML Infrastructure to design, build, and operate production-grade inference infrastructure powering the SambaNova serving stack on the Reconfigurable Dataflow Unit (RDU).

You will own end-to-end systems from request scheduling to accuracy infrastructure, collaborating with ML, compiler, runtime, and product teams to ship inference features from prototype to production.

Qualifications

• Experience designing and maintaining production services with strict latency, throughput, and availability requirements.
• Working knowledge of modern LLM inference techniques and familiarity with open-source serving stacks such as vLLM, TensorRT-LLM, or SGLang.
• Experience collaborating across teams to deliver complex, system-level engineering solutions.
• 5+ years of industry experience building and operating large-scale distributed systems, ideally in ML serving.
• B.S. in Computer Science, Electrical Engineering, or related field.
• Strong software engineering fundamentals: algorithms, data structures, concurrency, and systems design.
• Proficiency in Python.

Responsibilities

• Design and operate production-grade inference infrastructure powering SambaNova’s serving stack on RDU.
• Own end-to-end systems from request scheduling to accuracy infrastructure.
• Collaborate with ML, compiler, runtime, and product teams to ship inference features.
• Integrate SambaNova with vLLM and adjacent serving frameworks.
• Contribute to technical design discussions and code reviews.

Skills

LLM inference

Python

Distributed systems

Algorithms

Data structures

Concurrency

Systems design

Cross-team collaboration

Education

B.S. in CS/EE or related field

Tools

vLLM

TensorRT-LLM

SGLang

Job description

• The ML Infrastructure team builds and operates the inference stack that serves SambaNova’s models on RDU accelerators, from request scheduling and caching through the public APIs in SambaStack and SambaCloud
• We take inference techniques like speculative decoding, constrained decoding, and long-context serving from prototype to production, and own the accuracy infrastructure that gates every feature we ship
• We work alongside the ML, compiler, runtime, and product teams, since most of what we build touches all four
• The Senior Software Engineer, ML Infrastructure will be responsible for designing, building, and operating the production-grade inference infrastructure that powers SambaNova’s serving stack on our Reconfigurable Dataflow Unit (RDU) architecture
• SambaNova is an inference-first company, and this role sits at the heart of that mission: turning state-of-the-art inference techniques into reliable, high-throughput, low-latency services exposed to customers through SambaStack and SambaCloud
• The engineer will own end-to-end systems spanning request scheduling, advanced decoding algorithms, caching layers, API surfaces, and the accuracy infrastructure that keeps the stack trustworthy
• This role partners closely with ML, compiler, runtime, and product teams to ship inference features from prototype to production
• Design and productionize advanced inference techniques on RDU to optimize for performance and cost. Key areas include speculative decoding, constrained decoding, function/tool calling, prompt caching, and long-context inference
• Own SambaNova’s integration with vLLM and adjacent serving frameworks, adapting them to RDU’s architecture
• Own the public inference API surface exposed through SambaStack and SambaCloud
• Build and maintain the accuracy verification and regression infrastructure that gates every inference feature shipped to customers
• Partner with ML, compiler, runtime, and product teams to take inference features from prototype to production
• Contribute to technical design discussions, code reviews, and architectural decisions as a senior individual contributor

Benefits

• Flexible PTO in US
• Parental Leave
• Benefits (medical, dental and vision)
• Flexible Spending Accounts
• 401k/Pensions
• Gym Access
• Flexible Working Hours

Experience designing and maintaining production services with strict latency, throughput, and availability requirementsWorking knowledge of modern LLM inference techniques and familiarity with open-source serving stacks such as vLLM, TensorRT-LLM, or SGLangExperience collaborating across teams to deliver complex, system-level engineering solutions5+ years of industry experience building and operating large-scale distributed systems, ideally in ML servingB.S. in Computer Science, Electrical Engineering, or related fieldStrong software engineering fundamentals: algorithms, data structures, concurrency, and systems designProficiency in Python