Applied AI Evaluation Engineer – Benchmark & Innovate

Mistral; Paris; On-site; EUR 70,000 - 110,000; Full time

Benefits offered by this job

Benefits package

Job summary

Mistral is seeking an Evaluation Engineer to own end-to-end evals, working with customers, industry experts, and cross‑functional teams to define high‑impact AI use cases. You will design benchmarks, build real-software evals, and push for robust, reusable evaluation frameworks across models and competitors.

You will source data, score outcomes, and present findings to senior leaders, with opportunities to influence go-to-market evidence and future r&d directions.

Qualifications

• 2+ years of industry experience or a PhD in a relevant field.
• Software engineering or research background with reusable code.
• Experience working with customers, industry experts, or teams.
• Experience building with LLMs and handling large datasets.
• Strong English communication and ability to present to leaders.

Responsibilities

• Run discovery with customers and teams to identify high-value use cases.
• Design benchmarks for industry use cases and frontier capabilities.
• Build evals where agents work in real software and ensure reliability over time.
• Stress-test evals to uncover flaws before deployment.
• Source data with vendors or create realistic proxies when data cannot leave the customer.

Skills

Python programming

LLM knowledge

Data handling

Customer collaboration

Communications

Education

PhD

Tools

CAD tools

Python

Job description

Mistral is seeking an Evaluation Engineer to own end-to-end evals, working with customers, industry experts, and cross‑functional teams to define high‑impact AI use cases. You will design benchmarks, build real-software evals, and push for robust, reusable evaluation frameworks across models and competitors.

You will source data, score outcomes, and present findings to senior leaders, with opportunities to influence go-to-market evidence and future r&d directions.