AI Engineer (Harness)

Dex; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

Dex partners with a London-based sovereign AI infrastructure company to build and evaluate autonomous agent systems. You will own the measurement layer, designing experiments, calibrating LLM judges, and ensuring robust scoring across non-deterministic workflows.

The role emphasizes rigorous statistical methods, bias mitigation, and tooling with Python APIs to orchestrate agents and memory. Collaboration with defence-tech veterans is expected.

Qualifications

• Experience applying statistics to ML systems and experimental design.
• Proven ability to evaluate non-deterministic agent systems across repeated runs.
• Hands-on with LLM-as-a-judge pipelines against human ground truth.

Responsibilities

• Build and maintain Python evaluation harnesses scoring task completion, trajectory quality, tool-use correctness, cost, latency, and variance across repeated runs of non-deterministic agents.
• Apply statistical methods including hypothesis testing, confidence intervals, bootstrapping, and power analysis to reliably distinguish genuine performance gains from random variance.
• Develop, audit, and calibrate LLM-as-a-judge pipelines against human annotations, systematically detecting and mitigating biases in position, verbosity, and self-preference.
• Create golden evaluation datasets, scoring rubrics, and annotation guidelines from scratch while managing inter-annotator agreement and label noise.
• Run systematic root-cause analyses on agent failure modes and translate findings into architectural improvements across agent scaffolding, context management, retry logic, and safety guardrails.

Skills

Statistics & experimental design

Non-deterministic agent evaluation

LLM-as-a-judge pipelines

Python engineering

Tools

Python

LLM APIs

Job description

This role is with one of Dex's trusted partner companies. We work closely with their teams to truly understand their culture, goals, and what they're looking for, so we can match you with the right opportunity and give you context about the role before you commit to a process.

The role

When AI agents operate inside sovereign enclaves for defence, intelligence, and regulated enterprise, an unnoticed regression masquerading as an improvement is catastrophic. This London-based sovereign AI infrastructure company builds secure execution environments for national security, healthcare, and critical infrastructure to run autonomous systems on sensitive, air-gapped data. Backed by tens of millions in Series A funding from leading global venture firms and run by defence tech veterans, they are establishing the production standards for mission-critical agent deployment.

You will own the measurement layer: the Python evaluation harnesses that determine whether complex, non-deterministic agent workflows are genuinely improving or simply gaming metrics. This is not prompt engineering or subjective vibe-checking; it is rigorous experimental design, building calibrated LLM judges, isolating signal from probabilistic noise across repeated runs, and turning failure analyses into concrete upgrades across agent scaffolding, tool loops, and guardrails.

The work

• Build and maintain the core Python evaluation harnesses scoring task completion, trajectory quality, tool-use correctness, cost, latency, and variance across repeated runs of non-deterministic agents.
• Apply statistical methods including hypothesis testing, confidence intervals, bootstrapping, and power analysis to reliably distinguish genuine performance gains from random variance.
• Develop, audit, and calibrate LLM-as-a-judge pipelines against human annotations, systematically detecting and mitigating biases in position, verbosity, and self-preference.
• Create golden evaluation datasets, scoring rubrics, and annotation guidelines from scratch while managing inter-annotator agreement and label noise.
• Run systematic root-cause analyses on agent failure modes and translate your findings into architectural improvements across agent scaffolding, context management, retry logic, and safety guardrails.

What You Bring

• Strong background in statistics and experimental design applied to probabilistic machine learning systems, with the ability to design outcome-driven metrics and detect benchmark gaming.
• Proven track record evaluating multi-step, non-deterministic agent systems across repeated runs, including systematic error analysis and failure clustering.
• Hands-on experience developing and calibrating LLM-as-a-judge pipelines against human ground truth, with deep attention to position, verbosity, and self-preference biases.
• Production-grade Python engineering skills working directly with LLM APIs, structured outputs, tool orchestration, retrieval, and agent memory architectures.