Agentic Evaluation Engineer

Callosum; Greater London; On-site; GBP 90,000 - 150,000; Full time

Benefits offered by this job

Competitive Salary

Equity & Ownership

Private healthcare

Visa sponsorship and relocation

London office

Job summary

Callosum, the Intelligent Systems Company in London, is hiring an Agentic Evaluation Researcher to design benchmarks and evaluation tools for agentic AI across heterogeneous models and hardware.

You will develop rigorous measures, ensure validity, and translate findings into actionable guidance for product teams. The role offers visa sponsorship and a London office for in-person collaboration.

Qualifications

• You can design experiments to settle open questions and produce results others can build on.
• Hands-on experience with LLMs in agentic settings: multi-step tasks, tool use, long horizons, and failure modes.
• Statistical hypotheses, uncertainty quantification, and comparative analyses that isolate variables.
• Strong engineering: you write code to run experiments in a large shared codebase.
• Clear communication: translate research findings into actionable guidance for teams.

Responsibilities

• Design benchmarks and evaluation suites for agentic behavior across multi-turn tasks and non-stationary environments.
• Develop the methodology and the harness: power analysis, variance, contamination, and construct validity.
• Be the independent voice on the quality of results; perform sanity checks and red-team evaluations.
• Transform raw traces into structured evidence, including failure taxonomies and behavioral signatures.
• Predict performance from partial evidence and define capabilities needed for tasks.
• Build durable evaluation and observability infrastructure the company relies on.

Skills

Evidence you can run research

LLMs in agentic settings

Statistical discipline

Strong engineering

Strong communication skills

Job description

Callosum, the Intelligent Systems Company in London, is hiring an Agentic Evaluation Researcher to design benchmarks and evaluation tools for agentic AI across heterogeneous models and hardware.

You will develop rigorous measures, ensure validity, and translate findings into actionable guidance for product teams. The role offers visa sponsorship and a London office for in-person collaboration.