Senior ML Infrastructure Engineer

SambaNova; San Jose (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Medical Insurance

Health Savings Account (HSA)

Dental Insurance

Vision Insurance

Short Term Disability

Long Term Disability

Basic Life Insurance

Voluntary Life Insurance

AD&D Insurance

Flexible Spending Account (FSA)

Headspace Subscription

Gympass+ Membership

One Medical Membership

Employee Assistance Program

Equity

Job summary

SambaNova seeks an engineer to design and operate production-grade inference infrastructure on RDU architecture, optimizing performance and cost while sustaining reliability.

You will own the public API surface and build accuracy verification infrastructure to ensure high-throughput services, with a strong emphasis on scalable distributed systems and modern LLM serving techniques.

Qualifications

• Requires a B.S. in Computer Science or a related field and 5+ years of experience building large-scale distributed systems.
• Must be proficient in Python and familiar with modern LLM inference techniques and serving stacks like vLLM.

Responsibilities

• Design and operate production-grade inference infrastructure on RDU architecture to optimize performance and cost.
• Own the public API surface and build accuracy verification infrastructure to ensure reliable, high-throughput services.

Skills

Distributed Systems

ML Serving

Python

Systems Design

LLM Inference

vLLM

Algorithms

Data Structures

Concurrency

API Design

Speculative Decoding

Constrained Decoding

Prompt Caching

Long-context Inference

TensorRT-LLM

SGLang

Education

B.S. in Computer Science or related field

Job description

Design and operate production-grade inference infrastructure on RDU architecture to optimize performance and cost. Own the public API surface and build accuracy verification infrastructure to ensure reliable, high-throughput services.

Requirements: Requires a B.S. in Computer Science or a related field and 5+ years of experience building large-scale distributed systems. Must be proficient in Python and familiar with modern LLM inference techniques and serving stacks like vLLM.

Key Skills: Distributed Systems, ML Serving, Python, Systems Design, LLM Inference, vLLM, Algorithms, Data Structures, Concurrency, API Design, Speculative Decoding, Constrained Decoding, Prompt Caching, Long-context Inference, TensorRT-LLM, SGLang

Benefits: Medical Insurance, Health Savings Account (HSA), Dental Insurance, Vision Insurance, Short Term Disability, Long Term Disability, Basic Life Insurance, Voluntary Life Insurance, AD&D Insurance, Flexible Spending Account (FSA), Headspace Subscription, Gympass+ Membership, One Medical Membership, Employee Assistance Program, Equity