AI Benchmarking Engineer for Cross-Stack Evaluation

Callosum Technologies Ltd.; Greater London; Hybrid; GBP 75,000 - 120,000; Full time

Benefits offered by this job

Visa sponsorship

Relocation benefits

Job summary

Callosum Technologies Ltd. is hiring for a research-driven role in London to build and run a unified benchmarking system for evaluating agentic and algorithmic LLM workflows.

You will design tasks, collect real commits/traces, and ensure robust, contamination-free evaluation with reproducible results. The role requires a PhD or equivalent track record, strong Python, and experience with sandboxed/distributed execution.

Qualifications

• PhD in computer science, ML, or related field, or equivalent research track record.
• Authorship of benchmark/evaluation papers at NeurIPS/ICML/ACL is preferred.
• Understanding of evaluation pitfalls: contamination, overfitting, irreproducible results.
• Strong Python programming and experience with sandboxed/distributed execution and CI.
• Hands-on experience building or evaluating multi-step LLM systems.

Responsibilities

• Design and build a unified benchmarking system for evaluating agentic and algorithmic workloads.
• Mine real commits/traces, run sandboxed grading, and build task suites reflecting real agentic work.
• Enforce controls against contamination, overfitting to benchmarks, and metric gaming.
• Compare algorithmic and agentic approaches honestly, focusing on resolved-task quality.
• Lead external benchmark co-publications and publish results.
• Incorporate results into which approach ships and QA quality claims.

Skills

Benchmark evaluation

Sandboxed execution

Python

Agentic/LLM eval

Publications in benchmarks

Education

PhD in computer science or related field

Tools

Python

Job description

Callosum Technologies Ltd. is hiring for a research-driven role in London to build and run a unified benchmarking system for evaluating agentic and algorithmic LLM workflows.

You will design tasks, collect real commits/traces, and ensure robust, contamination-free evaluation with reproducible results. The role requires a PhD or equivalent track record, strong Python, and experience with sandboxed/distributed execution.