Senior ML Engineer, LLM Inference Optimization

Nebius Group; Amsterdam; Hybrid; EUR 120,000 - 180,000; Full time

Benefits offered by this job

Competitive compensation

Career growth and learning opportuni…

Flexibility and ownership

Collaborative and innovative culture

Impactful AI projects

International environment

Job summary

Nebius is seeking a Senior Machine Learning Engineer to own model and endpoint optimization across production deployments. You will work on model internals, quantization, and inference efficiency, collaborating with kernel and platform engineers to improve latency, throughput, and cost per token while preserving quality.

You will design experiments, build reproducible benchmarks, and drive safe production rollouts for multi-node GPU deployments, with a focus on performance, reliability, and

Qualifications

• Strong engineering skills in Python and PyTorch.
• Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference.
• Practical knowledge of at least one modern inference stack (e.g., vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, KServe).
• A strong understanding of transformer inference bottlenecks involvingKV cache, attention, memory bandwidth, batching, parallelism, and long-context serving.
• Ability to quantify trade-offs among latency, throughput, quality, utilization, and cost.
• Experience designing or optimizing distributed inference system architecture (data routing, distributed scheduling, PDD, multi-node inference, or expert parallelism).
• Strong communication and collaboration skills across research, kernel, infrastructure, product, and customer teams.

Responsibilities

• Own optimization projects for specific model families, customer endpoints, or serving backends.
• Compare inference engines and recommend serving configurations suited to individual workloads.
• Diagnose model-quality and performance regressions during production rollouts.
• Improve LLM and VLM endpoint latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.
• Deploy, configure, benchmark, and extend engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo.
• Develop production model-compression workflows covering quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.
• Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving.
• For prefill–decode disaggregation (PDD), evaluate KV-cache transfer, worker placement, and capacity balancing to determine which workloads benefit from the architecture.
• Design and improve LLM request routers and scheduling policies that balance worker utilization, queueing, request characteristics, and KV-cache locality while meeting latency and reliability targets.
• Scale dense and mixture-of-experts inference across multiple GPU nodes and tune parallelism with hardware topology considerations.
• Build reproducible benchmark harnesses measuring TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token.
• Collaborate with GPU kernel and platform engineers to trace bottlenecks across model code, kernels, runtime, scheduler, gateway, and cluster layers.
• Produce design documents, performance reports, rollout plans, and customer-facing technical explanations.

Skills

Python

PyTorch

LLM deployment

Distributed inference

Performance optimization

Benchmarking

Communication

Tools

vLLM

SGLang

TensorRT-LLM

Triton Inference Server

NVIDIA Dynamo

Ray Serve

KServe

CUDA

Job description

Nebius is seeking a Senior Machine Learning Engineer to own model and endpoint optimization across production deployments. You will work on model internals, quantization, and inference efficiency, collaborating with kernel and platform engineers to improve latency, throughput, and cost per token while preserving quality.

You will design experiments, build reproducible benchmarks, and drive safe production rollouts for multi-node GPU deployments, with a focus on performance, reliability, and