Senior ML Inference Engineer — Optimize Throughput (Remote)

Creandum; France; Remote; EUR 107,000 - 160,000; Full time

Benefits offered by this job

Equity options

Remote-first global environment

Job summary

Cast AI is seeking an experienced engineer to optimize inference stacks, focusing on throughput, latency, and cost per token. You’ll tune loaders, schedulers, and KV caches, shaping the technical direction for scalable multi-GPU deployments.

Join a remote-first, globally distributed team that ships fast, with equity options and a strong emphasis on data-driven performance and reproducible experiments.

Qualifications

• You've built real ML systems for 5+ years, and your portfolio shows depth in inference or training infrastructure, not just model training notebooks.
• Your Python is production-grade: services, not scripts.
• You've run vLLM, SGLang, or TensorRT-LLM for real, and you have a working mental model of why an engine performs the way it does on a given GPU.
• You've measured quantization regressions, not just compression ratios.
• Multi-GPU and multi-node setups hold no fear: tensor and expert parallelism, collective communication, and their practical failure modes.
• You know where the bytes live (page caches, O_DIRECT, network filesystems under terabyte-scale reads) and how fast they move.
• You instrument before you optimize, and you can tell a real win from a benchmark artifact.
• A wide mandate excites you more than it unsettles you.

Responsibilities

• Push throughput. Continuous batching, chunked prefill, speculative decoding with draft models, CUDA-graph capture, and scheduler tuning across the engine fleet. Find the ceiling on each GPU SKU, then raise it.
• Cut latency. Attack TTFT and inter-token latency separately; profile to find the actual bottleneck (compute, memory bandwidth, scheduling, storage, or network) and fix it, not the one you assumed.
• Get more out of the KV cache. Hierarchical and paged attention, prefix caching on heavy-reuse coding traffic, eviction policies, quantized KV.
• Kill cold starts. Weight loaders (O_DIRECT, fastsafetensors-class), JIT and compile caches, shared-filesystem read paths. A 1.5 TB checkpoint has no business taking an hour to reach serving.
• Quantize without regressing quality. INT8/INT4, FP8/FP4 across weights, activations, and KV, measured on real workloads, not just perplexity benchmarks.
• Scale across nodes. Distributed inference topologies, network-aware placement, prefill/decode disaggregation, checkpointing strategies that don't bottleneck on storage or interconnect.
• Set the technical direction. Decide what we benchmark, what we adopt, and what we build ourselves. Bring the team along with strong writeups and reproducible experiments.

Skills

ML systems

Python production

vLLM/SGLang/TensorRT-LLM

Quantization

Multi-GPU/Multi-node

Benchmarking

Tools

SGLang

vLLM

TensorRT-LLM

Python

PyTorch

CUDA tooling

Kubernetes

ClickHouse

OpenTelemetry

Prometheus

Grafana

GitLab CI

Job description

Cast AI is seeking an experienced engineer to optimize inference stacks, focusing on throughput, latency, and cost per token. You’ll tune loaders, schedulers, and KV caches, shaping the technical direction for scalable multi-GPU deployments.

Join a remote-first, globally distributed team that ships fast, with equity options and a strong emphasis on data-driven performance and reproducible experiments.