Staff Research Engineer: Agentic Evaluation & Benchmarks

Callosum Technologies Ltd.; Greater London; Hybrid; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Competitive salary

Equity & Ownership

Private healthcare

Visa sponsorship & relocation

London office in-person work

Job summary

Callosum Technologies Ltd. is building the next generation of AI systems through application-aware orchestration across heterogeneous models and hardware.

This role focuses on agentic evaluation, establishing robust benchmarks and evidence-based measures to locate failures and improve designs. You will design and run experiments, quantify uncertainty, and publish results to guide the company’s future products.

Qualifications

• Evidence that you can run research of your own and design experiments.
• Hands-on experience with LLMs in agentic settings and multi-step tasks.
• Strong statistical discipline and uncertainty quantification.
• Strong software engineering skills and ability to contribute to a large codebase.
• Excellent communication to translate research into actionable guidance.

Responsibilities

• Design benchmarks and evaluation suites for agentic behaviour across multi-turn tasks.
• Develop methodological tooling: power analysis, variance across runs, validity checks.
• Act as independent voice on result quality; perform red-teaming of evaluations.
• Convert raw traces into structured evidence and failure taxonomies.
• Predict system capabilities from partial evidence and framework tasks.

Skills

Research capability

LLMs in agentic settings

Statistics

Engineering

Communication

Benchmarks

Empirical methods

Automated grading

Publications

Job description

Callosum Technologies Ltd. is building the next generation of AI systems through application-aware orchestration across heterogeneous models and hardware.

This role focuses on agentic evaluation, establishing robust benchmarks and evidence-based measures to locate failures and improve designs. You will design and run experiments, quantify uncertainty, and publish results to guide the company’s future products.