Applied AI Evaluation Engineer: LLM Benchmarks & Pipelines

Mistral; Paris; On-site; EUR 90,000 - 130,000; Full time

Benefits offered by this job

Healthcare coverage

Parental leave

Retirement plans

Relocation support

Wellness programs

Meal and transportation allowances

Job summary

Mistral is seeking an Evaluation Engineer to shape how enterprise customers measure and trust AI solutions. You will design evaluation systems at the intersection of research, engineering, and product, collaborating with clients and internal stakeholders to move projects from concept to production with clear accountability.

You will work on frameworks for LLM performance, scalable pipelines, and custom evaluation suites, partnering with research and product teams to deliver measurable

Qualifications

• 3+ years of experience evaluating ML/LLM/agentic systems.
• Strong Python coding skills.
• Deep understanding of ML concepts and algorithms, esp. LLMs.
• Experience communicating technical topics to mixed audiences.
• Familiarity with evaluation frameworks like LM Eval Harness or OpenAI Evals; contributions to related projects.
• Experience with ML libraries such as PyTorch and HuggingFace Transformers.

Responsibilities

• Design frameworks to evaluate LLM performance across customer use cases.
• Build scalable evaluation infrastructure and pipelines.
• Develop new evaluation methods for sector-specific capabilities.
• Collaborate with customers to create custom evaluation suites.
• Partner with research teams to translate evaluation insights into model improvements.
• Work with product teams to refine evaluation tooling based on feedback.
• Define and communicate clear success metrics for production-ready AI models.

Skills

Python coding

ML concepts

Communication

Evaluation frameworks

ML libraries (PyTorch, HuggingFace)

Tools

LM Eval Harness

OpenAI Evals

Job description

Mistral is seeking an Evaluation Engineer to shape how enterprise customers measure and trust AI solutions. You will design evaluation systems at the intersection of research, engineering, and product, collaborating with clients and internal stakeholders to move projects from concept to production with clear accountability.

You will work on frameworks for LLM performance, scalable pipelines, and custom evaluation suites, partnering with research and product teams to deliver measurable