AI Benchmarking Architect

AI Startups UK; Greater London; Hybrid; GBP 120,000 - 180,000; Full time

Benefits offered by this job

Competitive salary

Equity & ownership

Private healthcare

Visa sponsorship and relocation

London office

Job summary

Callosum is building a unified benchmarking system for evaluating agentic and algorithmic AI work. The role focuses on reproducible, contamination‑controlled evaluation across motifs and decomposition strategies, grounded in real execution traces and sandboxed grading.

You will architect and implement pipelines that measure task success, quality and robustness, influencing which approaches ship and how we communicate proof points to customers.

Qualifications

• PhD or equivalent in computer science, ML, or related field.
• Authorship or co-authorship of benchmarks or evaluation papers at NeurIPS/ICML/ICLR/ACL.
• Strong understanding of evaluation pitfalls: contamination, overfitting, irreproducibility.
• Engineering ability: Python expertise, sandboxed/distributed execution, CI.
• Hands-on experience evaluating agentic or multi-step LLM systems.

Responsibilities

• Design and build a unified system for evaluating agentic and algorithmic solutions across workloads, tracking task success, quality and robustness.
• Mine real commits and traces, run sandboxed execution grading, and build task suites reflecting real agentic work.
• Enforce controls against contamination, overfitting, and metric gaming; keep baselines stable and reproducible.
• Compare algorithmic and agentic approaches; determine which to ship based on resolved-task quality.
• Lead external benchmark co-publications and ensure alignment with peer and customer review.
• Incorporate results into product decisions and customer communications.

Job description

Callosum is building a unified benchmarking system for evaluating agentic and algorithmic AI work. The role focuses on reproducible, contamination‑controlled evaluation across motifs and decomposition strategies, grounded in real execution traces and sandboxed grading.

You will architect and implement pipelines that measure task success, quality and robustness, influencing which approaches ship and how we communicate proof points to customers.