Agentic Evaluation Engineer — Benchmarks & Systems

AI Startups UK; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Competitive salary

Equity ownership

Private healthcare

Visa sponsorship & relocation

Job summary

Callosum in London is hiring for an expert to lead agentic evaluation. You’ll build principled benchmarks, evaluate heterogeneous models and hardware, and ensure results stand up to scrutiny across tasks and generations of silicon.

Your work informs product directions and system design in a fast-moving AI stack. You will design experiments, quantify uncertainty, and partner with engineering to deploy durable observability.

Qualifications

• Evidence you can run research of your own: design experiments and produce results others can build on.
• Deep hands-on experience with LLMs in agentic settings: multi-step tasks, tool use, long horizons.
• Statistical discipline with hypotheses, uncertainty quantification, and rigorous comparisons.
• Strong engineering to implement experiments inside a large shared codebase.
• Clear, structured communication to guide teams and write for a wider audience.

Responsibilities

• Design benchmarks and evaluation suites for agentic behaviour – multi-turn, long-horizon tasks.
• Develop methodology: statistical power, variance across runs, contamination and validity.
• Be the independent voice on results; red-team evaluations and sanity checks.
• Turn raw traces into structured evidence and failure taxonomies.
• Move from measurement to prediction by defining each model's capability profile.
• Build durable evaluation and observability infrastructure used company-wide.

Skills

Research design

Statistical analysis

Engineering

LLMs/agentic systems

Communication

Tools

LLMs

Experiment frameworks

Job description

Callosum in London is hiring for an expert to lead agentic evaluation. You’ll build principled benchmarks, evaluate heterogeneous models and hardware, and ensure results stand up to scrutiny across tasks and generations of silicon.

Your work informs product directions and system design in a fast-moving AI stack. You will design experiments, quantify uncertainty, and partner with engineering to deploy durable observability.