AI Engineer: Production ML & Evaluation

Mistral AI; Paris; Sur place; EUR 65 000 - 95 000; Plein temps

Résumé du poste

Mistral AI in Paris is seeking a software/ML engineer to join a product team, focusing on evaluating and improving AI-powered features across search, chat, documents, and audio. You’ll define what good looks like, measure it, run experiments, and ship improvements that raise quality, latency, safety, and reliability.

You will design prompts, orchestrations, and system prompts, run A/B tests, build observability and release processes, and collaborate with Science to push production-ready AI

Qualifications

• 3–4 years of experience in ML or software engineering with AI production exposure.
• Strong TypeScript or Python skills.
• Production LLM experience: prompts, tool calls, system prompts.
• Hands-on with evals and A/B testing.
• Observability experience: logging, tracing, dashboards, alerting.
• Product mindset: form hypotheses, run experiments, ship.
• Clear communication and autonomy with production impact.

Responsabilités

• Design and run evaluations for your product area: reference tests, heuristics, model-graded checks tailored to search relevance, chat quality, document understanding, or audio performance.
• Define and track metrics that matter: task success, helpfulness, latency, safety flags, cost.
• Own prompt and orchestration design: write, test, and iterate on prompts and system prompts as a core part of your work.
• Run A/B tests on prompts, models, and configurations; analyze results; make rollout or rollback decisions from data.
• Set up observability for LLM calls: structured logging, tracing, dashboards.
• Operate model releases: canary and shadow traffic, sign-offs, SLO-based rollback criteria.
• Improve core behaviors in your product area: memory policies, retrieval quality, routing, tool-call reliability.
• Create templates and documentation so other teams can author evals and ship safely.
• Partner with Science to diagnose regressions and lead post-mortems.

Connaissances

TypeScript

Python

Production ML

Eval & A/B testing

Observability

Product mindset

Prompt engineering

Outils

Logging

Dashboards

Description du poste

Mistral AI in Paris is seeking a software/ML engineer to join a product team, focusing on evaluating and improving AI-powered features across search, chat, documents, and audio. You’ll define what good looks like, measure it, run experiments, and ship improvements that raise quality, latency, safety, and reliability.

You will design prompts, orchestrations, and system prompts, run A/B tests, build observability and release processes, and collaborate with Science to push production-ready AI