Agentic Evaluation Engineer, Research Scientist

Callosum; Greater London; On-site; GBP 120,000 - 180,000; Full time

Benefits offered by this job

Private healthcare

Visa sponsorship

Relocation benefits

London office

Job summary

Callosum in London seeks a research engineer to advance agentic evaluation for AI systems. You will design benchmarks, develop evaluation suites, and turn traces into actionable evidence, measuring what models can actually do across multi-step tasks and tool use.

You will also quantify uncertainty, reduce reliance on single scores, and help shape product decisions with rigorous, reproducible results. Ideal candidates have hands-on experience with LLMs in agentic settings, strong statistical

Qualifications

• Experience designing and evaluating agentic AI systems.
• Demonstrated ability to build rigorous, reproducible experiments.

Responsibilities

• Design benchmarks and evaluation suites for agentic behaviour - multi-turn, long-horizon, tool-using in non-stationary environments.
• Develop the methodology and the harness: power analysis, run-to-run variance, contamination control and construct validity.
• Be the independent voice on results quality, perform red-team evaluation, and implement sanity checks.
• Turn raw traces into structured evidence, including failure taxonomies and behavioural signatures.
• Predict system capabilities from partial evidence and define relevant capability profiles.
• Build durable evaluation and observability infrastructure for the company.

Skills

Agentic evaluation

LLMs usage

Statistical methods

Engineering

Communication

Education

PhD in ML

Tools

Python

PyTorch

SQL

Job description

Callosum in London seeks a research engineer to advance agentic evaluation for AI systems. You will design benchmarks, develop evaluation suites, and turn traces into actionable evidence, measuring what models can actually do across multi-step tasks and tool use.

You will also quantify uncertainty, reduce reliance on single scores, and help shape product decisions with rigorous, reproducible results. Ideal candidates have hands-on experience with LLMs in agentic settings, strong statistical