LLM Inference Performance Engineer

Xapply; San Francisco (CA); On-site; USD 170,000 - 230,000; Full time

Benefits offered by this job

Competitive compensation and equity

Medical, dental, vision insurance (US)

Flexible PTO including Winter Break

Parental leave

Carrot stipend for family-building

401(k) (US)

Exposure to ML startups

Job summary

Baseten is seeking an Software Engineer focusing on inference performance to accelerate AI workloads at scale. You will optimize runtime internals, quantization, decoding, and KV-cache management across GPUs, improving speed and cost efficiency for customers.

You will work on cutting-edge baseten inference stack, benchmark across architectures, and contribute to open-source engines. A strong background in GPU/LLM optimization is essential.

Qualifications

• Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, Mathematics, or related field.
• Experience with Python or C++ programming languages.
• Familiarity with LLM optimization techniques (quantization, speculative decoding, continuous batching).
• Strong familiarity with ML libraries, especially PyTorch, TensorRT, or TensorRT-LLM.
• Demonstrated interest and experience in LLMs.
• Deep understanding of GPU architecture.

Responsibilities

• Implement and productionize cutting-edge inference techniques, working deep in runtime internals.
• Profile and optimize inference end to end, from kernel launch to routing and cache-aware scheduling.
• Turn performance into cost savings by improving tokens per GPU-hour, utilization, and latency/throughput tradeoffs.
• Bring up and tune new model architectures on new hardware quickly.
• Build benchmarking frameworks that measure real-world performance across model architectures and hardware.
• Contribute upstream to open-source inference engines (vLLM, SGLang, TensorRT-LLM) and collaborate with teams to ship wins.

Skills

Python

C++

PyTorch

TensorRT

TensorRT-LLM

Quantization

Speculative Decoding

Continuous Batching

GPU Architecture

CUDA

Triton

CUTLASS

Education

Bachelor's/Master's/Ph.D. in CS/Engineering/Math

Tools

TensorRT

Triton

CUTLASS

vLLM

SGLang

TensorRT-LLM

Job description

Baseten is seeking an Software Engineer focusing on inference performance to accelerate AI workloads at scale. You will optimize runtime internals, quantization, decoding, and KV-cache management across GPUs, improving speed and cost efficiency for customers.

You will work on cutting-edge baseten inference stack, benchmark across architectures, and contribute to open-source engines. A strong background in GPU/LLM optimization is essential.