Director, AI Agent Behavior & Evaluation Systems

Blue Yonder; Paris; Sur place; EUR 140 000 - 200 000; Plein temps

Résumé du poste

Blue Yonder’s Autonomy Labs seeks a deeply technical Director of Model Behavior & Evaluation Systems to own behavioral quality for LLM agents. You’ll define what good looks like, set release gates, and build feedback loops from traces, SME review, and telemetry into product improvements.

You will lead through systems, standards, and people, guiding senior engineers while owning execution and roadmap decisions in a real-world supply chain context.

Qualifications

• Led technical work on LLM products, AI agents, or model behavior.
• Built or guided evaluation systems for open-ended LLM behavior.
• Strong depth with tool-calling, function calling, or enterprise automation.
• Fluent with Python, PyTorch, and modern LLM stacks.
• Experience partnering with ML teams on post-training workflows.
• Ability to translate user needs into shipped product behavior.

Responsabilités

• Own the behavioral quality bar for Blue Yonder’s LLM agents across workflows.
• Lead evaluation governance, release gates, and regression coverage.
• Define launch criteria: correctness, tool-use, escalation quality, and trust.
• Set direction for behavior/eval infra and telemetry analysis.
• Turn traces, feedback, and failures into data for improvements.
• Partner with RL/post-training teams to shape data and experiments.
• Collaborate with product, domain, and engineering teams for durable coverage.

Connaissances

Technical leadership

LLM product quality

Cross-functional collaboration

Stakeholder communication

Formation

Bachelor’s degree or higher in CS/ML/related field

Outils

Python

PyTorch

Hugging Face Transformers

Hugging Face Datasets

OpenAI Agents SDK

Langfuse

NVIDIA NeMo RL

Description du poste

Blue Yonder’s Autonomy Labs seeks a deeply technical Director of Model Behavior & Evaluation Systems to own behavioral quality for LLM agents. You’ll define what good looks like, set release gates, and build feedback loops from traces, SME review, and telemetry into product improvements.

You will lead through systems, standards, and people, guiding senior engineers while owning execution and roadmap decisions in a real-world supply chain context.