Senior IA Inference Engineer (x/f/m)

Doctolib; Paris; On-site; EUR 90,000 - 140,000; Full time

Job summary

Doctolib in Paris is seeking a Senior Inference Engineer to join the ML Platform (TIGER) team. You will own the self-hosted LLM serving platform powering our AI-driven healthcare products, from Consultation Assistant to Health Companion, and scale models across our GPU fleet.

You will build the shared foundations for AI features, deploy and tune open-weight models, ensure security and compliance, and drive performance and cost efficiency across Europe.

Qualifications

• 5+ years of software or computer engineering experience with hands-on production experience with at least one LLM serving engine.
• Proficient in Python and Terraform; solid understanding of transformer inference mechanics.
• Experience running GPU workloads on Kubernetes or a managed Ray / Anyscale platform; performance engineering mindset.
• Comfortable owning production reliability: observability, incident response, and on-call.
• Fluent in English.

Responsibilities

• Build the shared LLM serving foundations with MLOps engineers - define the serving stack, GPU capacity planning, and deployment automation for open-weight models on Ray Serve / Anyscale, moving toward Kubernetes-based orchestration.
• Own the model registry and promotion path - establish versioning, validation, and production promotion; benchmark performance and output quality; maintain the curated model catalog with the Agents Platform team.
• Optimize inference performance and efficiency - operate and tune serving engines to reduce latency and cost across the GPU fleet, using batching, caching, quantization, and parallelism.
• Own production reliability and disaster recovery - define SLOs, run load tests before launches, implement safe rollouts, rollbacks, and autoscaling, and participate in on-call.
• Make serving performance and costs measurable - build Datadog dashboards, alerting, and cost reporting to track latency, GPU utilization, and inference spend.
• Keep the inference stack secure, compliant, and evolving - collaborate with Security and Privacy on data residency and auditability; assess new models and engines and GPU generations.

Skills

Python

Terraform

Kubernetes

Performance engineering

Observability

English fluency

Tools

Ray Serve

Anyscale

Triton Inference Server

vLLM

TensorRT-LLM

MLflow

KServe

NVIDIA Dynamo

Job description

Set a new pulse for healthcare!

We are looking for a Senior Inference Engineer to join the ML Platform (TIGER) team in AI & Machine Learning .

Your mission will be to power Doctolib's AI-driven healthcare products - from Consultation Assistant to Phone Assistant and Health Companion - by owning the self-hosted LLM serving platform that keeps the most sensitive patient data within infrastructure Doctolib controls. You will build and operate the shared foundations that every AI feature team relies on: deploying, tuning, and scaling open-weight models across our GPU fleet, contributing directly to faster, safer, and more cost-efficient AI for care teams and patients across Europe.

Working in the tech team at Doctolib means building innovative products and features to improve the daily lives of care teams and patients.

What you'll do

Your responsibilities include but are not limited to:

• Build the shared LLM serving foundations with MLOps engineers - define the serving stack, GPU capacity planning, and deployment automation for open-weight models (Gemma, Mistral, Llama, Qwen, and their fine-tuned variants) on Ray Serve / Anyscale, while shaping the evolution toward Kubernetes-based orchestration
• Own the model registry and promotion path - establish clear processes for versioning, validating, and promoting models into production, benchmark performance and output quality under realistic workloads, and maintain the curated model catalog together with the Agents Platform team
• Optimize inference performance and efficiency - operate and tune serving engines (vLLM, Triton Inference Server, TensorRT-LLM) to reduce time-to-first-token, inter-token latency, and cost per token through batching, caching, quantization, and parallelism strategies across our GPU fleet (L4 to H200)
• Own production reliability and disaster recovery - define SLOs, run load tests before launches, implement safe rollouts, rollbacks, and autoscaling, and participate in on-call and incident response
• Make serving performance and costs measurable - build Datadog dashboards, alerting, and cost reporting to track latency, GPU utilization, and inference spend, and identify opportunities to improve resource efficiency
• Keep the inference stack secure, compliant, and evolving - work with Security and Privacy on network isolation, data residency, and auditability, while continuously assessing new models, serving engines (such as SGLang), and GPU generations

Who you are

Before you read on: if you don't have the exact profile described below, but you feel this job description matches your skill set, we still encourage you to apply.

You'll be a great fit if you:

• Have 5+ years of software or computer engineering experience, including hands-on production experience with at least one LLM serving engine (vLLM, SGLang, TensorRT-LLM, TGI, or Triton) - ideally having chosen, introduced, tuned, or scaled one in production, not just run it with defaults
• Are proficient in Python and Terraform, with a solid understanding of transformer inference mechanics: prefill vs. decode, KV cache, batching strategies, and why a model is compute- or memory-bound on a given GPU
• Have strong experience running GPU workloads on Kubernetes or a managed Ray / Anyscale platform, and a performance engineering mindset - you design reproducible benchmarks, read profiler output, and back decisions with numbers
• Are comfortable owning production reliability: observability, incident response, and on-call are a natural part of the role
• Have excellent communication and collaboration skills, high agency, a pragmatic build-vs-buy approach, and are comfortable working in a scope that is still taking shape
• Are fluent in English

It would be fantastic if you:

• Have experience with distributed serving frameworks (KServe, llm-d, NVIDIA Dynamo), including disaggregated prefill/decode or KV-cache-aware routing, or with serving Mixture-of-Experts, multimodal models, or many LoRA adapters on shared base models
• Have GPU kernel experience or deep profiling skills (CUDA, Triton, Nsight), or have contributed to open-source inference projects
• Have experience in regulated environments (healthcare, finance, public sector) and with frameworks such as GDPR, HDS, ISO 27001, or SOC 2

Our stack

• Programming languages: Python, plus comfort reading/debugging a systems language (C++, Rust, or Go)
• Cloud providers: GCP / AWS / Azure
• Serving orchestration: Ray Serve / Anyscale (GCP), with Kubernetes (GKE) + GPU scheduling as an emerging parallel target
• LLM serving engines (in production): vLLM, Triton Inference Server, TensorRT-LLM (also evaluating SGLang and other emerging engines)
• Models: Self-hosted open-weight LLMs (Gemma, Mistral, Llama, Qwen), fine-tuned in-house
• Experiment tracking & model registry: MLflow
• GenAI gateway: LiteLLM
• LLM evaluation & observability: Braintrust
• GPUs: NVIDIA L4, A10G, L40S, RTX 6000 Pro, A100 (40GB/80GB), H200

Co