Staff AI Engineer: Build Production ML Evaluation Tools

Doctolib

Paris

On-site

EUR 90,000 - 130,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Paid vacation 25 days
RTT (reduction of working time)
Mental health services
Work from abroad up to 10 days per año
Lunch vouchers
Transport reimbursement 50%
DoctoGrowth program
ParentCare program
Disability support

Job summary

Doctolib is hiring for a senior software/ML engineer to help design and operate advanced evaluation tooling for AI products. You will collaborate with product, design, and business teams to assess current evaluation quality and implement improvements across APIs, services, and CI/CD integrations.

You will design strategies combining deterministic checks, LLM-based judgment, and human review, and build production sampling and distribution monitoring capabilities.

Qualifications

  • Have 6+ years of experience as a software engineer, with significant Python experience, or as a machine learning engineer.
  • Experience building or operating evaluation for LLM-powered or agentic systems.
  • Understand and promote software development best practices.
  • Product-oriented, able to work with multiple teams and drive adoption.

Responsibilities

  • Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations).
  • Collaborate with product teams to assess current evaluation methods and define improvements.
  • Design evaluation strategies for agentic systems, combining deterministic checks and LLM-based review.
  • Build capabilities for production sampling, dataset representativeness monitoring, and test generation.
  • Define quality metrics, representative datasets, and reliable quality gates across the lifecycle.
  • Create reusable, self-service platform capabilities and measure adoption and coverage.
  • Contribute to cross-functional initiatives shaping Doctolib's AI platform and best practices.

Skills

Python
LLM evaluation
ML engineering
English communication

Tools

CI/CD
APIs

Job description

Doctolib is hiring for a senior software/ML engineer to help design and operate advanced evaluation tooling for AI products. You will collaborate with product, design, and business teams to assess current evaluation quality and implement improvements across APIs, services, and CI/CD integrations.

You will design strategies combining deterministic checks, LLM-based judgment, and human review, and build production sampling and distribution monitoring capabilities.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Engineer — Evaluation & Quality for AI Products
Staff AI Engineer — Evaluation & Quality for AI Products

Doctolib • Paris

On-site
EUR 90,000 - 130,000
Health insurance
Paid vacation
Mental health & coaching services
+2
Staff AI Engineer: Build Safe Healthcare AI Eval Tools
Staff AI Engineer: Build Safe Healthcare AI Eval Tools

Doctolib • Levallois-Perret

On-site
EUR 100,000 - 160,000
Health insurance
Vacation days: 25 days + RTT
Mental health and coaching services
+7
Staff AI Platform Engineer — Evaluation & Tools (Healthcare)
Staff AI Platform Engineer — Evaluation & Tools (Healthcare)

Doctolib • Paris

On-site
EUR 120,000 - 170,000
Health insurance
Vacation days (25)
Mental health support
+7
Senior Staff ML Engineer — Healthcare AI & LLM Strategy
Senior Staff ML Engineer — Healthcare AI & LLM Strategy

Doctolib • Levallois-Perret

Hybrid
EUR 150,000 - 190,000
Health insurance
Vacation days
Mental health & coaching
+5
Senior ML Engineer: Applied AI & LLMs for Scaled Care
Senior ML Engineer: Applied AI & LLMs for Scaled Care

Doctolib • Levallois-Perret

Hybrid
EUR 120,000 - 190,000
Health insurance
Paid vacation
RTT days
+6
Senior ML Engineer, AI Health Orchestration & Safe LLMs
Senior ML Engineer, AI Health Orchestration & Safe LLMs

Doctolib • Paris

Hybrid
EUR 110,000 - 150,000
Health insurance for you and family
25 days paid vacation + RTT
Lunch vouchers (Swile)
Senior ML Engineer — AI Evaluation & Safe Agentic Systems
Senior ML Engineer — AI Evaluation & Safe Agentic Systems

EngineersOfAI • Paris

Hybrid
EUR 70,000 - 95,000
Free health insurance for you and your children
Parent Care Program
Free mental health services
+2
Staff AI Engineer — Agent Evaluation Platform
Staff AI Engineer — Agent Evaluation Platform

Doctolib (France) • Paris, Levallois-Perret

On-site
EUR 120,000 - 180,000
Health insurance
Vacation days
Lunch vouchers
+2
Senior ML Engineer - Applied AI & LLMs for Patient Care
Senior ML Engineer - Applied AI & LLMs for Patient Care

EngineersOfAI • Paris

Hybrid
EUR 60,000 - 90,000
Senior AI Engineer - Health Search & Recommendations
Senior AI Engineer - Health Search & Recommendations

Doctolib • Levallois-Perret

On-site
EUR 120,000 - 170,000
Free comprehensive health insurance
Lunch vouchers
Transit subsidy