Senior LLM Inference Optimization Engineer

Fdetimes; Palo Alto (CA); On-site; USD 195,000 - 262,000; Full time

Benefits offered by this job

Health insurance

401(k) match

Parental leave

Remote work reimbursement

Insurance coverage

Job summary

Nebius, a Nasdaq-listed AI cloud leader, seeks a Senior Machine Learning Engineer to optimize model and endpoint performance from artifacts to production deployment. You will address model internals, quantization, KV-cache, and inference engines, ensuring latency, throughput, and cost targets while maintaining quality and reliability.

You will collaborate with kernel and platform engineers to diagnose bottlenecks, compare configurations, and validate improvements through reproducible benchmarks

Qualifications

• Strong engineering skills in Python and PyTorch.
• Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference.
• Practical knowledge of at least one modern inference stack (e.g., vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, KServe).
• Strong understanding of transformer inference bottlenecks (KV cache, attention, memory bandwidth, batching, parallelism, long-context serving).
• Ability to quantify trade-offs among latency, throughput, quality, utilization, and cost.
• Experience designing or optimizing distributed inference system architecture (request routing, distributed scheduling, PDD, multi-node inference, expert parallelism).
• Strong communication and collaboration across research, kernel, infrastructure, product, and customer teams.

Responsibilities

• Own optimization projects for specific model families, customer endpoints, or serving backends.
• Compare inference engines and recommend practical serving configurations.
• Diagnose model-quality and performance regressions during production rollouts.
• Improve LLM and VLM endpoint latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.
• Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo.
• Develop production model-compression workflows covering quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.
• Implement speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.
• Design and improve LLM request routers and scheduling policies balancing worker utilization and latency targets.
• Scale dense and mixture-of-experts inference across multiple GPU nodes with proper parallelism and topology.
• Build reproducible benchmark harnesses measuring TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.
• Collaborate with GPU kernel and platform engineers to trace bottlenecks across model code and cluster layers.
• Produce design documents, performance reports, rollout plans, and customer-facing technical explanations.

Skills

Python

PyTorch

LLM deployment

Inference optimization

Distributed inference

Communication skills

Tools

vLLM

SGLang

TensorRT-LLM

Triton Inference Server

NVIDIA Dynamo

Ray Serve

KServe

Job description

Nebius, a Nasdaq-listed AI cloud leader, seeks a Senior Machine Learning Engineer to optimize model and endpoint performance from artifacts to production deployment. You will address model internals, quantization, KV-cache, and inference engines, ensuring latency, throughput, and cost targets while maintaining quality and reliability.

You will collaborate with kernel and platform engineers to diagnose bottlenecks, compare configurations, and validate improvements through reproducible benchmarks