Benchmarking Engineer for Agentic AI Systems (London)

Callosum; Greater London; On-site; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Equity & Ownership

Private healthcare

Visa sponsorship and relocation

London office-based

Job summary

Callosum in London is building a unified benchmarking system for evaluating agentic and algorithmic AI solutions. You will design a harness to measure task success, quality, and robustness across motifs, agent topologies, and decomposition strategies, grounded in real commits and sandboxed grading.

This research hire builds and curates the evaluation framework, delivering proof points for customers and contributing to co-published benchmarks.

Qualifications

• PhD in computer science or equivalent research track record.
• Authorship or co-authorship of a benchmark or evaluation paper at a recognised venue.
• Strong understanding of evaluation pitfalls like contamination and overfitting.

Responsibilities

• Design, build, and curate a unified benchmarking harness for agentic and algorithmic solutions.
• Lead external benchmark co-publications and ensure reproducibility and transparency.

Skills

Python

Sandboxed execution

Distributed systems

Research track record

Agentic/LLM evaluation

Education

PhD in computer science

Equivalent research track record

Tools

GitHub

CI/CD

Job description

Callosum in London is building a unified benchmarking system for evaluating agentic and algorithmic AI solutions. You will design a harness to measure task success, quality, and robustness across motifs, agent topologies, and decomposition strategies, grounded in real commits and sandboxed grading.

This research hire builds and curates the evaluation framework, delivering proof points for customers and contributing to co-published benchmarks.