AI Evaluation Engineer: LLM Judging & Agent Metrics

Dex; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

Dex partners with a London-based sovereign AI infrastructure company to build and evaluate autonomous agent systems. You will own the measurement layer, designing experiments, calibrating LLM judges, and ensuring robust scoring across non-deterministic workflows.

The role emphasizes rigorous statistical methods, bias mitigation, and tooling with Python APIs to orchestrate agents and memory. Collaboration with defence-tech veterans is expected.

Qualifications

• Experience applying statistics to ML systems and experimental design.
• Proven ability to evaluate non-deterministic agent systems across repeated runs.
• Hands-on with LLM-as-a-judge pipelines against human ground truth.

Responsibilities

• Build and maintain Python evaluation harnesses scoring task completion, trajectory quality, tool-use correctness, cost, latency, and variance across repeated runs of non-deterministic agents.
• Apply statistical methods including hypothesis testing, confidence intervals, bootstrapping, and power analysis to reliably distinguish genuine performance gains from random variance.
• Develop, audit, and calibrate LLM-as-a-judge pipelines against human annotations, systematically detecting and mitigating biases in position, verbosity, and self-preference.
• Create golden evaluation datasets, scoring rubrics, and annotation guidelines from scratch while managing inter-annotator agreement and label noise.
• Run systematic root-cause analyses on agent failure modes and translate findings into architectural improvements across agent scaffolding, context management, retry logic, and safety guardrails.

Skills

Statistics & experimental design

Non-deterministic agent evaluation

LLM-as-a-judge pipelines

Python engineering

Tools

Python

LLM APIs

Job description

Dex partners with a London-based sovereign AI infrastructure company to build and evaluate autonomous agent systems. You will own the measurement layer, designing experiments, calibrating LLM judges, and ensuring robust scoring across non-deterministic workflows.

The role emphasizes rigorous statistical methods, bias mitigation, and tooling with Python APIs to orchestrate agents and memory. Collaboration with defence-tech veterans is expected.