AI Product Engineer - Prompts & Evaluation

Mistral.ai; Paris; Sur place; EUR 70 000 - 100 000; Plein temps

Résumé du poste

Mistral.ai is seeking an AI quality engineer embedded in a product team to advance AI features across search, chat, documents, and audio. You will define what good looks like, measure it, run experiments, and ship improvements that affect quality, latency, safety, and reliability.

You will design evaluations, track key metrics, own prompts and system prompts, and run A/B tests to inform rollout decisions. Strong production ML experience and a product mindset are essential.

Qualifications

• 3–4 years of experience in ML or software engineering with AI/ML production exposure.
• Strong TypeScript or Python skills.
• Production LLM experience: prompts, tool calls, system prompts.
• Hands-on with evals and A/B testing; design metrics, not only run them.
• Comfortable implementing directly in production code, not only notebooks.
• Observability: logging, tracing, dashboards, alerts.
• Form hypotheses, run experiments, interpret results, ship.
• Clear communication, autonomous, production impact over experimentation for its own sake.

Responsabilités

• Design and run evaluations for product area: reference tests, heuristics, model-graded checks for search relevance, chat quality, document understanding, or audio performance.
• Define and track metrics: task success, helpfulness, hallucination proxies, safety signals, latency, cost.
• Own prompt and orchestration design: write, test, and iterate prompts and system prompts.
• Run A/B tests on prompts, models, configurations; analyze results and decide rollout/rollback.
• Set up observability for LLM calls: logging, tracing, dashboards, alerts.
• Manage model releases: canary and shadow traffic, sign-offs, SLO-based rollback criteria.
• Improve core behaviors: memory policies, routing, tool-call reliability, or retrieval quality.
• Create templates and documentation for evals and safe shipping.
• Partner with Science to diagnose regressions and lead post-mortems.

Connaissances

TypeScript

Python

Evals & A/B testing

Observability

Product mindset

Description du poste

Mistral.ai is seeking an AI quality engineer embedded in a product team to advance AI features across search, chat, documents, and audio. You will define what good looks like, measure it, run experiments, and ship improvements that affect quality, latency, safety, and reliability.

You will design evaluations, track key metrics, own prompts and system prompts, and run A/B tests to inform rollout decisions. Strong production ML experience and a product mindset are essential.