Staff AI Engineer: Build Safe Healthcare AI Eval Tools

Doctolib

Levallois-Perret

On-site

EUR 100,000 - 160,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Vacation days: 25 days + RTT
Mental health and coaching services
Work from abroad up to 10 days/year
Lunch vouchers (Swile)
Sport club membership subsidy
Transport subsidy 50%
DoctoGrowth plan enrollment
ParentCare Program: salary coverage in
Disability support remote policy & --

Job summary

Doctolib is seeking a Staff AI Software Engineer to join the EvalX AI Evaluation team in Levallois-Perret, helping AI product teams measure quality and reliability. You will build shared evaluation capabilities used in production across AI products for patients and clinicians.

You will work at the intersection of platform engineering and applied AI, designing evaluation strategies that combine deterministic checks, LLM-as-a-judge, and simulations, while enabling teams through reusable tooling

Qualifications

  • 6+ years of experience as a software engineer, with significant experience using Python; or as a machine learning engineer.
  • Experience building or operating evaluation for LLM-powered or agentic systems: datasets, LLM-as-a-judge, human review.
  • Understand and promote software development best practices.
  • Product-oriented and able to work directly with other teams to translate needs into solutions.
  • Influence engineering practices and drive change without hierarchical authority.
  • Comfortable speaking and writing in English in day-to-day work.

Responsibilities

  • Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations) used by AI product teams across Doctolib.
  • Collaborate with product teams to assess evaluation today, define improvements, and help them own evaluation quality.
  • Design evaluation strategies for agentic systems, combining deterministic checks, LLM-as-a-judge, simulation-based testing, and human review.
  • Build intelligent capabilities for production sampling, dataset representativeness monitoring, distribution-shift detection, and test case generation.
  • Help teams define meaningful quality metrics, representative datasets, and reliable quality gates throughout the development lifecycle.
  • Turn recurring product needs into reusable, self-service platform capabilities, and measure adoption and team autonomy.
  • Contribute to cross-functional initiatives that shape Doctolib's AI platform and best practices for agentic systems.

Skills

Python
Machine learning

Tools

APIs
CI/CD

Job description

Doctolib is seeking a Staff AI Software Engineer to join the EvalX AI Evaluation team in Levallois-Perret, helping AI product teams measure quality and reliability. You will build shared evaluation capabilities used in production across AI products for patients and clinicians.

You will work at the intersection of platform engineering and applied AI, designing evaluation strategies that combine deterministic checks, LLM-as-a-judge, and simulations, while enabling teams through reusable tooling

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Platform Engineer — Evaluation & Tools (Healthcare)
Staff AI Platform Engineer — Evaluation & Tools (Healthcare)

Doctolib • Paris

On-site
EUR 120,000 - 170,000
Health insurance
Vacation days (25)
Mental health support
+7
Staff AI Engineer — Evaluation & Quality for AI Products
Staff AI Engineer — Evaluation & Quality for AI Products

Doctolib • Paris

On-site
EUR 90,000 - 130,000
Health insurance
Paid vacation
Mental health & coaching services
+2
Staff AI Engineer — Agent Evaluation Platform
Staff AI Engineer — Agent Evaluation Platform

Doctolib (France) • Paris, Levallois-Perret

On-site
EUR 120,000 - 180,000
Health insurance
Vacation days
Lunch vouchers
+2
Staff AI Engineer: Build Production ML Evaluation Tools
Staff AI Engineer: Build Production ML Evaluation Tools

Doctolib • Paris

On-site
EUR 90,000 - 130,000
Health insurance
Paid vacation 25 days
RTT (reduction of working time)
+7
Senior AI Engineer — Architect of Healthcare AI Assistants
Senior AI Engineer — Architect of Healthcare AI Assistants

Doctolib • Nantes

On-site
EUR 90,000 - 130,000
Free comprehensive health insurance
ParentCare program
Mental health support
+6
Senior ML Engineer, AI Health Orchestration & Safe LLMs
Senior ML Engineer, AI Health Orchestration & Safe LLMs

Doctolib • Paris

Hybrid
EUR 110,000 - 150,000
Health insurance for you and family
25 days paid vacation + RTT
Lunch vouchers (Swile)
Senior AI Engineer - Health Search & Recommendations
Senior AI Engineer - Health Search & Recommendations

Doctolib • Levallois-Perret

On-site
EUR 120,000 - 170,000
Free comprehensive health insurance
Lunch vouchers
Transit subsidy
Staff AI Engineer - Agent Evaluation Platform (x/f/m)
Staff AI Engineer - Agent Evaluation Platform (x/f/m)

Doctolib • Levallois-Perret

On-site
EUR 100,000 - 160,000
Health insurance
Vacation days: 25 days + RTT
Mental health and coaching services
+7
Staff Applied AI Software Engineer (Python) – AI Evaluation (x/f/m)
Staff Applied AI Software Engineer (Python) – AI Evaluation (x/f/m)

Doctolib • Paris

On-site
EUR 120,000 - 170,000
Health insurance
Vacation days (25)
Mental health support
+7
Staff AI Engineer - Agent Evaluation Platform (x/f/m)
Staff AI Engineer - Agent Evaluation Platform (x/f/m)

Doctolib • Paris

On-site
EUR 90,000 - 130,000
Health insurance
Paid vacation
Mental health & coaching services
+2