Applied AI Evaluation Engineer: LLM Benchmarking & Readiness

Mistral.ai; Paris; On-site; EUR 60,000 - 90,000; Full time

Job summary

Mistral.ai is seeking an Evaluation Engineer to shape how enterprise customers measure and trust AI solutions. You will design evaluation frameworks, build scalable pipelines, and collaborate with clients and internal teams to drive production-grade AI models.

The role sits at the intersection of research, engineering, and solutions, contributing to model improvements across varied domains. The ideal candidate has 3+ years of experience evaluating ML/LLM systems, strong Python skills, and

Qualifications

• 3+ years of experience evaluating ML/LLM/agentic systems.
• Experience building AI/ML products with APIs or back-end systems.
• Strong Python coding skills.
• Deep understanding of ML concepts and algorithms for LLMs.
• Ability to communicate technical topics to technical and non-technical audiences.
• Familiarity with evaluation frameworks like LM Eval Harness or OpenAI Evals or related open-source projects.
• Experience with ML libraries such as PyTorch and HuggingFace Transformers.
• Collaborative, direct communicator who values outcomes and low-ego teamwork.

Responsibilities

• Design and implement frameworks to evaluate LLM performance across customer use cases.
• Build and maintain scalable evaluation infrastructure and pipelines.
• Develop new evaluation methods for sector-specific model capabilities.
• Collaborate with customers to create custom evaluation suites.
• Partner with research teams to translate evaluation insights into model improvements.
• Work with product teams to refine evaluation tooling based on user feedback.
• Define and communicate clear success metrics for production-ready AI models.

Skills

ML/LLM evaluation

APIs/Back-end

Python

ML concepts

Communication

Evaluation frameworks

PyTorch / Transformers

Teamwork

Tools

PyTorch

HuggingFace Transformers

Job description

Mistral.ai is seeking an Evaluation Engineer to shape how enterprise customers measure and trust AI solutions. You will design evaluation frameworks, build scalable pipelines, and collaborate with clients and internal teams to drive production-grade AI models.

The role sits at the intersection of research, engineering, and solutions, contributing to model improvements across varied domains. The ideal candidate has 3+ years of experience evaluating ML/LLM systems, strong Python skills, and