AI Evaluation Platform Engineer — Scale Research on GPU

Mistral; Paris; On-site; EUR 90,000 - 130,000; Full time

Benefits offered by this job

Healthcare coverage

Relocation support

Wellness programs

Meal and transportation allowances

Job summary

Mistral is seeking a Research Engineer for the Eval Platform to build the backbone that measures model quality, ensuring it's reliable, reproducible and fast. You will contribute to scalable evaluations across GPU clusters, delivering usable results to researchers.

You'll collaborate with researchers to transform new evaluation needs into robust tooling, with a focus on reproducibility, access, and trust in evaluation outcomes.

Qualifications

• Master's or PhD in Computer Science or equivalent experience.
• 4+ years building production-grade software, ideally ML codebases or distributed systems.
• Excellent Python and strong software-design instincts: testing, code review, CI/CD.
• Experience running workloads on GPU clusters (Slurm, Kubernetes, Ray).
• Familiarity with LLM inference and evaluation.
• A product mindset: researchers are your users.
• Self-starter, low-ego, collaborative.

Responsibilities

• Build systems that keep eval results reproducible and comparable over time, as models, benchmarks and code evolve.
• Run evaluations at scale across our GPU clusters, from model serving to scoring.
• Make eval results easy to access, explore and trust, through APIs and dashboards that researchers use every day.
• Catch broken or noisy evals before they mislead research decisions.
• Support evaluation of agentic, multi-turn and tool-using models.
• Work closely with researchers to turn new evaluation needs into robust, shared tooling.

Skills

Python

CI/CD

GPU clusters

LLM inference

Product mindset

Self-starter

Collaboration

Education

Master's or PhD in Computer Science

Tools

Slurm

Kubernetes

Ray

vLLM

Job description

Mistral is seeking a Research Engineer for the Eval Platform to build the backbone that measures model quality, ensuring it's reliable, reproducible and fast. You will contribute to scalable evaluations across GPU clusters, delivering usable results to researchers.

You'll collaborate with researchers to transform new evaluation needs into robust tooling, with a focus on reproducibility, access, and trust in evaluation outcomes.