Director of AI Agent Behavior & Evaluation

JDA Software; Paris; On-site; EUR 150,000 - 190,000; Full time

Job summary

Blue Yonder seeks a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioural quality system for its LLM agents. You will define what good means, establish release gates, and build feedback loops turning traces, SME review, telemetry, and eval failures into model improvements.

This director-level role requires leadership through systems, standards, and people, guiding senior engineers toward robust, scalable infrastructure while aligning with customer workflows in

Qualifications

• Experience leading technical work on LLM products, AI agents, model behaviour, model evaluation, post-training, AI alignment, or AI product quality.

Responsibilities

• Own the behavioural quality bar for Blue Yonder's LLM agents across customer-facing supply chain workflows.
• Build and lead the model behaviour and evaluation systems function across behaviour specs, eval governance, SME review, release gates, regression coverage, and launch readiness.
• Define launch criteria across operational correctness, tool-use accuracy, workflow completion, escalation quality, safe fallback behaviour, refusal quality, consistency, and customer trust.
• Establish evaluation authority so evals become release decision infrastructure, not just model-quality reporting.
• Set the technical direction for behaviour and eval infrastructure across Python eval harnesses, OpenAI Agents SDK workflows, Langfuse traces, LLM-as-judge workflows, deterministic checks, trace analysis, reward/report versioning, and model-candidate comparison.
• Convert model traces, tool-call failures, SME feedback, red-team findings, telemetry, and customer-facing failures into behaviour specs, eval requirements, training data needs, and model improvement priorities.
• Partner with the reinforcement learning and post-training organization to turn behaviour gaps into SFT data, preference data, reward criteria, curriculum, NeMo RL experiments, model-candidate decisions, and regression tests.
• Partner with workflow, data, product, and domain experts to turn supply chain workflow truth into durable scenario coverage, rubrics, synthetic scenarios, eval datasets, and training data requirements.
• Partner with agent architecture and product engineering teams to ensure prompts, tools, APIs, skills, system instructions, and product workflows express the intended model behaviour consistently.
• Review model traces, eval outputs, experiment summaries, dataset slices, reward reports, and post-training results closely enough to make informed launch and roadmap decisions.
• Own customer and user behaviour discovery for agent workflows: what users expect agents to do, explain, ask, verify, elevate, refuse, and act on.
• Lead red-teaming and behavioural risk programmes for hallucinated operational claims, incorrect tool use, overconfidence, poor escalation, prompt injection, unsafe recommendations, sycophancy, and unhelpful refusals.
• Define the operating cadence for behavioural quality reviews, model release readiness, model cards, issue triage, regression management, and post-launch behaviour monitoring.
• Build a high-performing team or function around model behaviour, evaluation governance, SME review systems, and behavioural launch quality.
• Communicate model behaviour strategy, quality tradeoffs, launch readiness, and residual risks clearly to executives, customers, product teams, engineering teams, and domain experts.

Skills

LLM products

AI agents

model behaviour

model evaluation

post-training

AI alignment

AI product quality

leadership

cross-functional collaboration

Tools

Python

PyTorch

Hugging Face Transformers

Hugging Face Datasets

OpenAI Agents SDK

Langfuse

LLM evaluation harnesses

tool-calling traces

model checkpoints

production observability

Job description

Blue Yonder seeks a deeply technical Director of Model Behavior & Evaluation Systems to own the behavioural quality system for its LLM agents. You will define what good means, establish release gates, and build feedback loops turning traces, SME review, telemetry, and eval failures into model improvements.

This director-level role requires leadership through systems, standards, and people, guiding senior engineers toward robust, scalable infrastructure while aligning with customer workflows in