Senior ML Engineer: AI Inference & Model Optimization

FDE Times; Dubai; Remote; AED 717,000 - 963,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Parental leave

Remote work reimbursement

Disability & life insurance

Job summary

Nebius Token Factory is seeking a Senior Machine Learning Engineer to own model and endpoint optimization from model artifacts through production deployment.

You will work on model internals, quantization and model compression, speculative decoding, KV-cache optimization, and inference engine tuning to improve latency, throughput, memory efficiency, GPU utilization, and cost per token while preserving quality.

Qualifications

• Strong engineering skills in Python and PyTorch.
• Hands-on experience deploying or optimizing LLM, VLM, or high-throughput transformer inference.
• Practical knowledge of at least one modern inference stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, Ray Serve, or KServe.
• A strong understanding of transformer inference bottlenecks involving KV cache, attention, memory bandwidth, batching, parallelism, and long-context serving.
• Ability to quantify trade-offs among latency, throughput, quality, utilization, and cost.
• Experience designing or optimizing distributed inference system architecture, with hands-on work in one or more areas such as request routing, distributed scheduling, PDD, multi-node inference, or expert parallelism.
• Strong communication skills and collaboration skills across research, kernel, infrastructure, product, and customer teams.

Responsibilities

• Own optimization projects for specific model families, customer endpoints, or serving backends.
• Compare inference engines and recommend practical serving configurations suited to individual workloads.
• Diagnose model-quality and performance regressions during production rollouts.
• Improve LLM and VLM endpoint latency, throughput, memory efficiency, GPU utilization, quality, and cost per token.
• Deploy, configure, benchmark, and extend inference engines such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, and NVIDIA Dynamo.
• Develop production model-compression workflows covering quantization, quantization-aware training, distillation, low-bit serving, and accuracy recovery.
• Implement or integrate speculative decoding, draft-model approaches, KV-cache optimization, prefix caching, chunked prefill, continuous batching, and disaggregated prefill/decode serving. For prefill/decode disaggregation (PDD), evaluate KV-cache transfer, worker placement, and capacity balancing to determine which workloads benefit from the architecture.
• Design and improve LLM request routers and scheduling policies that balance worker utilization queuing, request characteristics, and KV-cache locality while meeting latency and reliability targets
• Scale dense and mixture-of-experts inference across multiple GPU nodes. Select and tune tensor, pipeline, data, and expert parallelism - including wide expert parallelism (WideEP) - with attention to hardware topology, expert load balance, and communication overhead.
• Build reproducible benchmark harnesses measuring for TTFT, TPOT, tokens per second per GPU, p95/p99 latency, GPU memory, reliability, and cost per token. Validate architecture choices under representative traffic and consistent GPU budgets.
• Collaborate with GPU kernel engineers and platform engineers to trace bottlenecks across model code, kernels, runtime, scheduler, gateway, and cluster layers.
• Produce design documents, performance reports, rollout plans, and customer-facing technical explanations.

Skills

Python

PyTorch

LLM inference

Distributed systems

Performance tuning

Benchmarking

Communication

Tools

vLLM

SGLang

TensorRT-LLM

Triton Inference Server

NVIDIA Dynamo

Ray Serve

KServe

Job description

Nebius Token Factory is seeking a Senior Machine Learning Engineer to own model and endpoint optimization from model artifacts through production deployment.

You will work on model internals, quantization and model compression, speculative decoding, KV-cache optimization, and inference engine tuning to improve latency, throughput, memory efficiency, GPU utilization, and cost per token while preserving quality.