LLM Inference Performance Engineer

Emploive; New York (NY); On-site; USD 180,000 - 360,000; Full time

Benefits offered by this job

Equity

Insurance for dependents

Winter Break

Job summary

Baseten is seeking an Inference Performance Engineer to accelerate AI workloads in production. You will work across the stack—from the inference engine to routing—using techniques like quantization, speculative decoding, and KV-cache management.

You’ll shape latency, throughput, and cost for customers’ models in a fast-growing startup. You’ll collaborate on open-source engines (vLLM, SGLang, TensorRT-LLM) and partner with model, infra, and customer-facing teams to ship wins.

Qualifications

• Bachelor's, Master's, or Ph.D. in CS, Engineering, Mathematics, or related field.
• Experience with Python or C++ and ML libraries (PyTorch, TensorRT).
• Familiarity with LLM optimization techniques and GPU architecture.

Responsibilities

• Implement and productionize inference techniques in runtime internals (quantization, speculative decoding, KV-cache).
• Profile and optimize end-to-end inference, from kernel launches to routing and batching.
• Build benchmarking frameworks and contribute upstream to open-source inference engines.

Skills

Python

C++

LLM optimization

GPU architecture

PyTorch

TensorRT

Education

Bachelor's degree

Master's degree

PhD

Tools

vLLM

SGLang

TensorRT-LLM

Job description

Baseten is seeking an Inference Performance Engineer to accelerate AI workloads in production. You will work across the stack—from the inference engine to routing—using techniques like quantization, speculative decoding, and KV-cache management.

You’ll shape latency, throughput, and cost for customers’ models in a fast-growing startup. You’ll collaborate on open-source engines (vLLM, SGLang, TensorRT-LLM) and partner with model, infra, and customer-facing teams to ship wins.