Staff AI Engineer – Agent Evaluation Platform (x/f/m)

Doctolib

Paris

On-site

EUR 90,000 - 130,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
Paid vacation 25 days
RTT (reduction of working time)
Mental health services
Work from abroad up to 10 days per año
Lunch vouchers
Transport reimbursement 50%
DoctoGrowth program
ParentCare program
Disability support

Job summary

Doctolib is hiring for a senior software/ML engineer to help design and operate advanced evaluation tooling for AI products. You will collaborate with product, design, and business teams to assess current evaluation quality and implement improvements across APIs, services, and CI/CD integrations.

You will design strategies combining deterministic checks, LLM-based judgment, and human review, and build production sampling and distribution monitoring capabilities.

Qualifications

  • Have 6+ years of experience as a software engineer, with significant Python experience, or as a machine learning engineer.
  • Experience building or operating evaluation for LLM-powered or agentic systems.
  • Understand and promote software development best practices.
  • Product-oriented, able to work with multiple teams and drive adoption.

Responsibilities

  • Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations).
  • Collaborate with product teams to assess current evaluation methods and define improvements.
  • Design evaluation strategies for agentic systems, combining deterministic checks and LLM-based review.
  • Build capabilities for production sampling, dataset representativeness monitoring, and test generation.
  • Define quality metrics, representative datasets, and reliable quality gates across the lifecycle.
  • Create reusable, self-service platform capabilities and measure adoption and coverage.
  • Contribute to cross-functional initiatives shaping Doctolib's AI platform and best practices.

Skills

Python
LLM evaluation
ML engineering
English communication

Tools

CI/CD
APIs

Job description

Join a team of passionate and hardworking entrepreneurs to transform healthcare!

Working in the tech team at Doctolib involves building innovative products and features to improve the daily lives of care teams and patients. We work in feature teams in an agile environment, while collaborating with product, design, and business teams.

What you'll do
  • Design, ship, and operate production-grade evaluation tooling (APIs, services, CI/CD integrations) used by AI product teams across Doctolib
  • Work hands-on with product teams to assess how they evaluate today, define improvement plans, and help them own their evaluation quality
  • Design evaluation strategies for agentic systems, combining deterministic checks, LLM-as-a-judge, simulation-based testing, and human review
  • Build intelligent capabilities for production sampling, dataset representativeness monitoring, distribution-shift detection, and test case generation
  • Help teams define meaningful quality metrics, representative datasets, and reliable quality gates throughout the development lifecycle
  • Turn recurring product needs into reusable, self-service platform capabilities, and measure success through adoption, team autonomy, and evaluation coverage
  • Contribute to cross-functional initiatives that shape Doctolib's AI platform and best practices for building and evaluating agentic systems
Who you are
  • Have 6+ years of experience as a software engineer, with significant experience using Python; or as a machine learning engineer
  • Have built or operated evaluation for LLM-powered or agentic systems: datasets, LLM-as-a-judge, human review, production-driven evaluation
  • Understand and promote software development best practices
  • Are product-oriented and enjoy working directly with other teams, challenging assumptions, and turning complex needs into concrete solutions
  • Can influence engineering practices and drive change without hierarchical authority: your job is not done until your solution is adopted
  • Are comfortable speaking and writing in English in your day-to-day with the team
What we offer
  • Free comprehensive health insurance (basic package) for you and your children
  • 25 days of paid vacation per year, plus up to 14 days of RTT
  • Free mental health and coaching services through our partner Moka.care
  • Work from abroad for up to 10 days per year thanks to our flexibility days policy
  • Lunch vouchers (Swile card) worth €8.50 per working day, with €4.50 covered by Doctolib
  • A subsidy from the work council to refund part of the membership to a sport club or a creative class
  • 50% reimbursement of your public transport subscription
  • Enrollment in Doctolib's long-term employee value sharing plan called DoctoGrowth
  • ParentCare Program: Enjoy full salary coverage (100%) during your first month of birth leave, and 75% during the second, covered by Doctolib
  • For caregivers and workers with disabilities, a package including an adaptation of the remote policy, extra days off for medical reasons, and psychological support
The interview process
  • A first interview with a member of the Talent Acquisition team
  • A coding interview
  • An agentics system design
  • A behavioural interview
  • Reference check
Job details
  • Permanent position
  • Full Time
  • Workplace: Levallois
  • Start date : As soon as possible
Now, it would be fantastic if you:
  • Have experience with applied ML techniques such as embeddings, clustering, statistical sampling, and distribution monitoring
  • Have experience building developer platforms or internal tooling, or in a Forward Deployed Engineering role
  • Have experience in the medical domain or another regulated environment

If you would like to find out more about tech life at Doctolib, feel free to read our latest Medium blog articles!

Doctolib is an equal opportunity employer.

We don't just accept diversity at Doctolib, we respect and celebrate it!

The more diverse ideas are heard, the more our product will truly improve healthcare for all.

You are welcome to apply to Doctolib, regardless of your gender, religion, age, sexual orientation, ethnicity, disability, or place of origin.

If you have a disability, let us know if there's any way we can make the interview process smoother for you!

All the information transmitted via this form is processed by Doctolib for the purpose of managing applications. For more information on how Doctolib processes your application data, click here. If you wish to exercise your rights or if you have any questions about the processing of your data, you can write to us at hr.dataprivacy@doctolib.com.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Engineer - Agent Evaluation Platform (x/f/m)
Staff AI Engineer - Agent Evaluation Platform (x/f/m)

Doctolib • Levallois-Perret

On-site
EUR 100,000 - 160,000
Health insurance
Vacation days: 25 days + RTT
Mental health and coaching services
+7
Staff AI Engineer - Agent Evaluation Platform (x/f/m)
Staff AI Engineer - Agent Evaluation Platform (x/f/m)

Doctolib • Paris

On-site
EUR 90,000 - 130,000
Health insurance
Paid vacation
Mental health & coaching services
+2
Staff Applied AI Software Engineer (Python) – AI Evaluation (x/f/m)
Staff Applied AI Software Engineer (Python) – AI Evaluation (x/f/m)

Doctolib • Paris

On-site
EUR 120,000 - 170,000
Health insurance
Vacation days (25)
Mental health support
+7
Staff AI Engineer - Agent Evaluation Platform (x/f/m) Nouveau
Staff AI Engineer - Agent Evaluation Platform (x/f/m) Nouveau

Doctolib (France) • Paris, Levallois-Perret

On-site
EUR 120,000 - 180,000
Health insurance
Vacation days
Lunch vouchers
+2
Senior Software Engineer (Agentic AI) - Python
Senior Software Engineer (Agentic AI) - Python

Doctolib • Paris

On-site
EUR 85,000 - 125,000
Free comprehensive health insurance
Parent Care Program: extra leave
Free mental health and coaching via M*
+3
Senior Staff Machine Learning Engineer - Clinical - AI Teams (x/f/m)
Senior Staff Machine Learning Engineer - Clinical - AI Teams (x/f/m)

Doctolib • Levallois-Perret

Hybrid
EUR 150,000 - 190,000
Health insurance
Vacation days
Mental health & coaching
+5
Senior AI Engineer - Patient Team (x/f/m)
Senior AI Engineer - Patient Team (x/f/m)

Doctolib • Levallois-Perret

On-site
EUR 120,000 - 170,000
Free comprehensive health insurance
Lunch vouchers
Transit subsidy
Senior Machine Learning Engineer - Orchestration - Applied AI & LLMs (x/f/m)
Senior Machine Learning Engineer - Orchestration - Applied AI & LLMs (x/f/m)

Doctolib • Paris

Hybrid
EUR 110,000 - 150,000
Health insurance for you and family
25 days paid vacation + RTT
Lunch vouchers (Swile)
Principal Machine Learning Engineer - Applied AI & LLMs (x/f/m)
Principal Machine Learning Engineer - Applied AI & LLMs (x/f/m)

Doctolib • Paris

Hybrid
EUR 150,000 - 190,000
Health insurance
Vacation allowance
Mental health support
+6
Senior Agents Platform Engineer (x/f/m)
Senior Agents Platform Engineer (x/f/m)

Doctolib • Levallois-Perret

Hybrid
EUR 90,000 - 130,000
Free comprehensive health insurance
25 days vacation
Lunch vouchers
+2