Frontier AI Evaluation Engineer: Build Next-Gen Benchmarks

Mistral Ai; Paris; On-site; EUR 90,000 - 130,000; Full time

Job summary

Mistral Ai is seeking an Evaluation Engineer to own evals from start to finish, running workshops with customers and industry experts to identify the most impactful use cases. You will design benchmarks, source data, score outcomes, and run evaluations across new Mistral models and competitors.

You will collaborate with post-training teams to tighten gaps, increase challenge, and present findings to senior leaders for strategic decisions.

Qualifications

• 2+ years of industry experience (e.g., in software, AI, or consulting) or a PhD.
• A software engineering or research background, with experience writing code that others can build on.
• Experience working directly with customers, industry experts, or other teams (e.g., through consulting, solutions engineering, or research collaborations).
• Experience building something with LLMs.
• Comfort working with large datasets, including building and cleaning them.

Responsibilities

• Run discovery with customers, industry experts and teams across Mistral, then help decide which use cases are worth building first.
• Design new benchmarks for industry use cases and frontier capabilities, including how they're scored.
• Build evals where agents work inside real software (e.g., CAD and simulation tools), and run them on our models with checks that the results stay reliable over time.
• Stress-test existing evals to find flaws and bugs before we rely on them.
• Source data with expert vendors. Where customer data can't leave the customer, build realistic proxy or synthetic data instead.
• Write up and present what you find so leaders across Mistral can act on it.
• Help other teams build their own evals, with tools and guides they can reuse.

Skills

Python

Code reviews

LLM experience

Large datasets

Public speaking

Tools

Python

Job description

Mistral Ai is seeking an Evaluation Engineer to own evals from start to finish, running workshops with customers and industry experts to identify the most impactful use cases. You will design benchmarks, source data, score outcomes, and run evaluations across new Mistral models and competitors.

You will collaborate with post-training teams to tighten gaps, increase challenge, and present findings to senior leaders for strategic decisions.