Head of AI Behavior & Evaluation Systems

jda; Paris; Sur place; EUR 120 000 - 180 000; Plein temps

Résumé du poste

Blue Yonder seeks a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioral quality system for our LLM agents operating in supply chain workflows. You will define what good means for agents, establish release gates, and build feedback loops from traces, SME review, telemetry, and eval failures into improvements.

You will lead through systems, standards, people, and decisions, collaborating with senior engineers and product partners to shape the end-to-end

Qualifications

• Director-level technical leadership role in AI for supply chain.

Responsabilités

• Own the behavioral quality bar for Blue Yonder's LLM agents across customer-facing workflows.
• Build and lead the model behavior and evaluation systems function across specs, governance, release gates, regression coverage, and launch readiness.
• Define launch criteria across operational correctness, tool-use accuracy, workflow completion, escalation quality, safe fallback behavior, refusal quality, consistency, and customer trust.
• Establish evaluation authority so evals become release decision infrastructure, not just reporting.
• Set technical direction for behavior and eval infra across Python eval harnesses, SDK workflows, Langfuse traces, and deterministic checks.
• Convert traces, tool-call failures, SME feedback, red-team findings, telemetry, and customer-facing failures into behavior specs and training data needs.
• Partner with RL and post-training orgs to turn gaps into SFT data, preference data, reward criteria, curriculum, NeMo RL experiments, and regression tests.
• Partner with workflow, data teams to align on data-ready processes and tooling.

Connaissances

Leadership

System thinking

ML model evaluation

Stakeholder collaboration

Outils

PyTorch

Hugging Face Transformers

Langfuse

OpenAI Agents SDK

NVIDIA NeMo RL

LLM evaluation harnesses

Description du poste

Blue Yonder seeks a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioral quality system for our LLM agents operating in supply chain workflows. You will define what good means for agents, establish release gates, and build feedback loops from traces, SME review, telemetry, and eval failures into improvements.

You will lead through systems, standards, people, and decisions, collaborating with senior engineers and product partners to shape the end-to-end