Head of LLM Behavior & Evaluation Systems

Blue Yonder; Paris; Sur place; EUR 150 000 - 190 000; Plein temps

Résumé du poste

Blue Yonder is seeking a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioral quality system for its LLM agents. This leader will define what good behavior means, establish release gates, and build feedback loops from traces, SME review, telemetry, and eval failures into model improvements.

The role demands systems thinking, technical depth in the modern ML stack, and the ability to guide senior engineers and cross-functional partners toward robust, scalable

Qualifications

• Led technical work on LLM products, AI agents, or AI product quality.
• Built evaluation systems for open-ended LLM behavior, including rubrics and release gates.
• Hands-on depth with LLM tool calling, function calling, or API agents.
• Fluent in modern LLM stack: Python, PyTorch, Hugging Face ecosystems.
• Understand post-training workflows: supervised fine-tuning, reward modeling, RLHF/RLAIF.
• Capable of inspecting model traces, eval failures, telemetry, and data to guide fixes.
• Strong product judgment translating user needs into shipped behavior.

Responsabilités

• Own the behavioral quality bar for Blue Yonder's LLM agents across workflows.
• Define launch criteria across operational correctness, tool-use, and escalation quality.
• Establish evaluation authority and release gates for deployments.
• Set technical direction for eval infra: Python harnesses, Langfuse traces, API workflows.
• Convert traces, SME feedback, telemetry into specs, data needs, and improvements.
• Partner with RL and post-training org to shape data and experiments.
• Collaborate with domain experts to cover supply chain scenarios.
• Lead red-teaming and behavioral risk programs for unsafe behavior and misuses.
• Define cadence for behavioral reviews, model cards, and post-launch monitoring.
• Build a high-performing team around model behavior and evaluation governance.
• Communicate strategy, risks, and decisions to executives and teams.

Connaissances

LLM product leadership

Evaluation systems

Tool calling

Python stack

Post-training workflows

Failure analysis

Trace inspection

Product judgment

Cross-functional leadership

Launch readiness

Excellent communicator

Model specs

Senior teams

Outils

NVIDIA NeMo RL

OpenAI Agents SDK

Langfuse

Hugging Face Transformers

Hugging Face Datasets

PyTorch

Python

Description du poste

Blue Yonder is seeking a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioral quality system for its LLM agents. This leader will define what good behavior means, establish release gates, and build feedback loops from traces, SME review, telemetry, and eval failures into model improvements.

The role demands systems thinking, technical depth in the modern ML stack, and the ability to guide senior engineers and cross-functional partners toward robust, scalable