ML Systems Engineer, Inference (Fully Remote)

Runpod; United States; Remote; USD 150,000 - 220,000; Full time

Benefits offered by this job

Equity

Medical, dental, and vision

Remote-first

Home office stipend

Job summary

Runpod is hiring a seasoned systems engineer to optimize LLM inference performance in a remote-first environment. You will define measurement pipelines for throughput, latency, and cost per token, and build repeatable tooling for rigorous benchmarking.

You will profile the serving stack from scheduler and memory to GPU kernels, diagnose bottlenecks, and implement optimizations for large models on single and multi-node GPU deployments. Join a fast-moving AI infrastructure team.

Qualifications

• 5+ years of professional system engineering experience with hands-on work in vLLM, SGLang, or a comparable serving engine in production.
• Strong Python software engineering skills and understanding of what drives LLM inference performance: batching, memory, parallelism, and latency/throughput trade-offs.
• Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving, plus rigorous benchmarking and GPU profiling.

Responsibilities

• Define how we measure inference performance, including throughput, time to first token, inter-token latency, and cost per token, and build tooling for rigorous, repeatable measurements.
• Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect.
• Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments.

Skills

Python

LLM serving

Benchmarking

GPU profiling

Throughput optimization

Tools

vLLM

SGLang

Profiling tools

Job description

• Define how we measure inference performance, including throughput, time to first token, inter-token latency, and cost per token, and build tooling for rigorous, repeatable measurements.
• Profile and diagnose performance problems across the serving stack, from scheduling and memory management down to kernels and interconnect.
• Improve serving efficiency for large, state-of-the-art models on single-node and multi-node GPU deployments.

Requirements:

• 5+ years of professional system engineering experience with deep, hands-on experience in vLLM, SGLang, or a comparable serving engine in production.
• Strong software engineering skills in Python and a solid understanding of what drives LLM inference performance: batching, memory, parallelism, and latency/throughput trade-offs.
• Experience with modern inference optimization techniques such as quantization, speculative decoding, or distributed serving, plus rigor in benchmarking and GPU profiling.

What You'll Receive:

• Competitive base pay from $150,000 - $220,000, meaningful equity, and generous medical, dental, and vision plans.
• Flexible PTO, a $1,200 home office and equipment stipend, and the opportunity to join a passionate team on the cutting edge of AI infrastructure.
• Most roles are remote-first with inclusive, collaborative teams using Slack for internal communication.

Runpod is the AI Developer Cloud, providing a platform for over one million developers to experiment, train, fine-tune, deploy, and scale AI. We're a small, remote-first team that takes ownership seriously, moves fast, and has processed more than 20 billion inference requests.