Staff AI Engineer — Evaluation & Quality for AI Products

Doctolib

Paris

On-site

EUR 90,000 - 130,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health insurance
Paid vacation
Mental health & coaching services
Lunch vouchers
Transport subsidies

Job summary

Doctolib is seeking a Staff AI Software Engineer to join the AI Evaluation team (EvalX) and help measure, understand and improve the quality, reliability and safety of AI systems across product teams.

You will design production-grade evaluation tooling, collaborate with product/design/business teams, and build platform capabilities for evaluation at scale in a fast-paced healthcare tech environment.

Qualifications

  • 6+ years of software engineering experience, with substantial Python usage or ML engineering background.
  • Experience evaluating LLM-powered or agentic systems: datasets, LLM-as-a-judge, human review, production evaluation.
  • Ability to promote software development best practices and work with cross-functional teams.
  • Product-oriented mindset and ability to turn complex needs into concrete solutions.
  • Ability to influence engineering practices and drive adoption of solutions without hierarchical authority.
  • Comfortable speaking and writing in English with the team.

Responsibilities

  • Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations) used by AI product teams across Doctolib.
  • Work hands-on with product teams to assess how they evaluate today, define improvement plans, and help them own their evaluation quality.
  • Design evaluation strategies for agentic systems, combining deterministic checks, LLM-as-a-judge, simulation-based testing, and human review.
  • Build intelligent capabilities for production sampling, dataset representativeness monitoring, distribution-shift detection, and test case generation.
  • Help teams define meaningful quality metrics, representative datasets, and reliable quality gates throughout the development lifecycle.
  • Turn recurring product needs into reusable, self-service platform capabilities, and measure success through adoption, team autonomy, and evaluation coverage.
  • Contribute to cross-functional initiatives that shape Doctolib's AI platform and best practices for building and evaluating agentic systems.

Skills

Python
LLM evaluation
Software engineering practices
Product collaboration

Tools

APIs
CI/CD
Evaluation tooling

Job description

Doctolib is seeking a Staff AI Software Engineer to join the AI Evaluation team (EvalX) and help measure, understand and improve the quality, reliability and safety of AI systems across product teams.

You will design production-grade evaluation tooling, collaborate with product/design/business teams, and build platform capabilities for evaluation at scale in a fast-paced healthcare tech environment.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Platform Engineer — Evaluation & Tools (Healthcare)
Staff AI Platform Engineer — Evaluation & Tools (Healthcare)

Doctolib • Paris

On-site
EUR 120,000 - 170,000
Health insurance
Vacation days (25)
Mental health support
+7
Staff AI Engineer: Build Safe Healthcare AI Eval Tools
Staff AI Engineer: Build Safe Healthcare AI Eval Tools

Doctolib • Levallois-Perret

On-site
EUR 100,000 - 160,000
Health insurance
Vacation days: 25 days + RTT
Mental health and coaching services
+7
Staff AI Engineer: Build Production ML Evaluation Tools
Staff AI Engineer: Build Production ML Evaluation Tools

Doctolib • Paris

On-site
EUR 90,000 - 130,000
Health insurance
Paid vacation 25 days
RTT (reduction of working time)
+7
Staff AI Engineer — Agent Evaluation Platform
Staff AI Engineer — Agent Evaluation Platform

Doctolib (France) • Paris, Levallois-Perret

On-site
EUR 120,000 - 180,000
Health insurance
Vacation days
Lunch vouchers
+2
Senior ML Engineer — AI Evaluation & Safe Agentic Systems
Senior ML Engineer — AI Evaluation & Safe Agentic Systems

EngineersOfAI • Paris

Hybrid
EUR 70,000 - 95,000
Free health insurance for you and your children
Parent Care Program
Free mental health services
+2
Senior AI Engineer — Architect of Healthcare AI Assistants
Senior AI Engineer — Architect of Healthcare AI Assistants

Doctolib • Nantes

On-site
EUR 90,000 - 130,000
Free comprehensive health insurance
ParentCare program
Mental health support
+6
Senior Staff ML Engineer — Healthcare AI & LLM Strategy
Senior Staff ML Engineer — Healthcare AI & LLM Strategy

Doctolib • Levallois-Perret

Hybrid
EUR 150,000 - 190,000
Health insurance
Vacation days
Mental health & coaching
+5
Senior AI Engineer - Health Search & Recommendations
Senior AI Engineer - Health Search & Recommendations

Doctolib • Levallois-Perret

On-site
EUR 120,000 - 170,000
Free comprehensive health insurance
Lunch vouchers
Transit subsidy
Staff Applied AI Software Engineer (Python) – AI Evaluation (x/f/m)
Staff Applied AI Software Engineer (Python) – AI Evaluation (x/f/m)

Doctolib • Paris

On-site
EUR 120,000 - 170,000
Health insurance
Vacation days (25)
Mental health support
+7
Senior ML Engineer, AI Health Orchestration & Safe LLMs
Senior ML Engineer, AI Health Orchestration & Safe LLMs

Doctolib • Paris

Hybrid
EUR 110,000 - 150,000
Health insurance for you and family
25 days paid vacation + RTT
Lunch vouchers (Swile)