Inference Systems Engineer: Optimize AI Serving & Latency

Emploive; San Francisco (CA); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Flexible work

Adaption Passport

Lunch Stipend

Well-Being

Job summary

Emploive is seeking an experienced ML systems engineer to own the cost and performance of our inference stack, shaping how we serve models as workloads, traffic, and hardware evolve. You will partner with the engineers operating the serving fleet to optimize caching, batching, quantization, decoding, and kernel-level tuning.

You will implement profiling systems to show where time and memory are spent and tune routing between internal infrastructure and external providers using cost and capacity

Qualifications

• 5+ years in ML systems, inference infrastructure, or performance engineering.
• Deep understanding of model serving, including prefill and decode, memory bandwidth, batching, and concurrency.
• Production experience with serving engines such as vLLM, SGLang, or TensorRT-LLM.
• Strong Python skills and proficiency in C++, Rust, or another systems language.
• Experience with GPU performance, including CUDA, NCCL, mixed precision, memory layout, kernels, or quantization.

Responsibilities

• Improve throughput, cost, and tail latency through KV-cache management, continuous batching, speculative decoding, and quantization.
• Optimize long-context prefill and decode workloads based on real production traffic.
• Tune routing between our infrastructure and external providers based on cost, capacity, and performance.
• Work within serving engines such as vLLM, SGLang, and TensorRT-LLM, going below the framework when needed.
• Build profiling and measurement systems that show where time, memory, and compute are being spent.

Skills

Python

C++/Rust

CUDA

NCCL

Batching

ML systems

Performance engineering

Tools

vLLM

SGLang

TensorRT-LLM

Job description

Emploive is seeking an experienced ML systems engineer to own the cost and performance of our inference stack, shaping how we serve models as workloads, traffic, and hardware evolve. You will partner with the engineers operating the serving fleet to optimize caching, batching, quantization, decoding, and kernel-level tuning.

You will implement profiling systems to show where time and memory are spent and tune routing between internal infrastructure and external providers using cost and capacity