Applied AI Evaluation Engineer: LLM Evaluation Pipelines

Mistral AI; Paris; On-site; EUR 90,000 - 120,000; Full time

Benefits offered by this job

Healthcare coverage

Retirement plans

Relocation support

Job summary

Mistral AI in Paris seeks an Evaluation Engineer to shape how enterprises measure, trust, and deploy AI solutions. You will design evaluation systems that determine readiness for real business impact and work across research, engineering, and customer teams to drive improvements.

As part of the Applied AI team, you will build scalable evaluation pipelines, specify success metrics, and collaborate with customers to tailor evaluation suites to their needs.

Qualifications

• 3+ years of experience evaluating ML/LLM or agentic systems.
• Proven background building AI products with APIs or back-end systems.
• Strong Python coding skills.
• Solid understanding of ML concepts and algorithms as they relate to LLMs.
• Ability to explain technical topics to diverse audiences.
• Familiarity with evaluation frameworks like LM Eval Harness or OpenAI Evals or related OSS.
• Experience using ML libraries such as PyTorch and HuggingFace Transformers.

Responsibilities

• Design and implement frameworks to evaluate LLM performance across customer use cases.
• Build and maintain scalable evaluation infrastructure and pipelines.
• Develop new evaluation methods for sector-specific model capabilities.
• Collaborate with customers to create custom evaluation suites.
• Partner with research teams to translate evaluation insights into model improvements.
• Work with product teams to refine evaluation tooling based on user feedback.
• Define and communicate clear success metrics for production-ready models.

Skills

Python

ML concepts

API / backend

Communication

Tools

PyTorch

Transformers

LM Eval Harness

Job description

Mistral AI in Paris seeks an Evaluation Engineer to shape how enterprises measure, trust, and deploy AI solutions. You will design evaluation systems that determine readiness for real business impact and work across research, engineering, and customer teams to drive improvements.

As part of the Applied AI team, you will build scalable evaluation pipelines, specify success metrics, and collaborate with customers to tailor evaluation suites to their needs.