Inference Performance Engineer: Fast LLMs, Equity & PTO

Baseten; San Francisco (CA); On-site; USD 190,000 - 260,000; Full time

Benefits offered by this job

Competitive compensation including equ

US: 100% medical, dental, vision for员工

Flexible PTO including Winter Break

Paid parental leave

Fertility and family-building stipend

401(k)

Exposure to ML startups

Job summary

Baseten is hiring an inference performance engineer to accelerate AI workloads across the inference stack, from runtime internals to scheduling and routing. You will apply techniques like prefill, speculative decoding, and KV-cache management, shaping how fast customers' models run in production.

The role requires strong GPU knowledge, experience with PyTorch and TensorRT, and the ability to ship production-ready optimizations in a fast-paced startup.

Qualifications

• Bachelor's, Master's, or Ph.D. in CS/Engineering/Math or related field.
• Experience with Python or C++.
• Familiarity with LLM optimization techniques (e.g., quantization, speculative decoding).
• Strong familiarity with ML libraries, especially PyTorch, TensorRT.
• Demonstrated interest and experience in LLMs.
• Deep understanding of GPU architecture.

Responsibilities

• Implement and productionize cutting-edge inference techniques, working deep in runtime internals.
• Profile and optimize inference end to end, from kernel launch overhead to scheduling and cache-aware routing.
• Turn performance into cost savings by improving tokens per GPU-hour and latency/throughput tradeoffs.
• Bring up and tune new model architectures on new hardware quickly, often in the same week they're released.
• Build benchmarking frameworks that measure real-world performance across model architectures and hardware.
• Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM) and partner with cross-functional teams to ship wins.

Skills

Python

C++

LLM optimization

Profiling & optimization

GPU architecture

Education

BSc/MSc/PhD in CS/Engineering/Math

Tools

PyTorch

TensorRT

CUDA

Triton

Job description

Baseten is hiring an inference performance engineer to accelerate AI workloads across the inference stack, from runtime internals to scheduling and routing. You will apply techniques like prefill, speculative decoding, and KV-cache management, shaping how fast customers' models run in production.

The role requires strong GPU knowledge, experience with PyTorch and TensorRT, and the ability to ship production-ready optimizations in a fast-paced startup.