Evaluation Engineer, Applied AI: Build Next-Gen LLM Benchmarks

Wwshemi; Paris; Hybrid; EUR 70,000 - 110,000; Full time

Job summary

Mistral is building frontier LLM evals to shape the next generation of flagship models. As an Evaluation Engineer, you will own evals from start to finish, running workshops with customers and internal teams to identify critical industry use cases and design benchmarks to measure them inside real software.

You will source data, define scoring, and run benchmarks on our models and on competitors, then publish results that guide product and go-to-market decisions.

Qualifications

• 2+ years of industry experience (AI or software) or a PhD.
• Software engineering or research background with runnable code.
• Experience working with customers, industry experts, or teams.
• Experience building something with LLMs.
• Comfort with large datasets and data cleaning.

Responsibilities

• Run discovery with customers and teams to identify high-value use cases.
• Design benchmarks for industry use cases and frontier capabilities and how they’re scored.
• Build evals where agents work inside real software and run on our models and competitors.
• Stress-test evals, identify flaws, and iterate.
• Source data with vendors or create realistic proxy/synthetic data when needed.
• Document and present findings to leaders to inform decisions.
• Help other teams build their own evals with reusable tools.

Skills

Python

LLM evals

Customer-facing

Independent/work with ambiguity

Education

PhD or equivalent

Tools

Python

Job description

Mistral is building frontier LLM evals to shape the next generation of flagship models. As an Evaluation Engineer, you will own evals from start to finish, running workshops with customers and internal teams to identify critical industry use cases and design benchmarks to measure them inside real software.

You will source data, define scoring, and run benchmarks on our models and on competitors, then publish results that guide product and go-to-market decisions.