Senior ML Inference Systems Engineer (Remote-First)

Runpod; United States; Remote; USD 150,000 - 220,000; Full time

Benefits offered by this job

Equity

Medical, dental, and vision

Remote-first

Home office stipend

Job summary

Runpod is hiring a seasoned systems engineer to optimize LLM inference performance in a remote-first environment. You will define measurement pipelines for throughput, latency, and cost per token, and build repeatable tooling for rigorous benchmarking.

You will profile the serving stack from scheduler and memory to GPU kernels, diagnose bottlenecks, and implement optimizations for large models on single and multi-node GPU deployments. Join a fast-moving AI infrastructure team.

Qualifications

• 5+ years of professional system engineering experience with hands-on work in vLLM, SGLang, or a comparable serving engine in production.
• Strong Python software engineering skills and understanding of what drives LLM inference performance: batching, memory, parallelism, and latency/throughput trade-offs.
• Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving, plus rigorous benchmarking and GPU profiling.

Responsibilities

• Define how we measure inference performance, including throughput, time to first token, inter-token latency, and cost per token, and build tooling for rigorous, repeatable measurements.
• Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect.
• Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments.

Skills

Python

LLM serving

Benchmarking

GPU profiling

Throughput optimization

Tools

vLLM

SGLang

Profiling tools

Job description

Runpod is hiring a seasoned systems engineer to optimize LLM inference performance in a remote-first environment. You will define measurement pipelines for throughput, latency, and cost per token, and build repeatable tooling for rigorous benchmarking.

You will profile the serving stack from scheduler and memory to GPU kernels, diagnose bottlenecks, and implement optimizations for large models on single and multi-node GPU deployments. Join a fast-moving AI infrastructure team.