AI Benchmarking Architect

AI Startups UK; Greater London; Híbrido; GBP 120 000 - 180 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Competitive salary

Equity & ownership

Private healthcare

Visa sponsorship and relocation

London office

Resumo da oferta

Callosum is building a unified benchmarking system for evaluating agentic and algorithmic AI work. The role focuses on reproducible, contamination‑controlled evaluation across motifs and decomposition strategies, grounded in real execution traces and sandboxed grading.

You will architect and implement pipelines that measure task success, quality and robustness, influencing which approaches ship and how we communicate proof points to customers.

Qualificações

• PhD or equivalent in computer science, ML, or related field.
• Authorship or co-authorship of benchmarks or evaluation papers at NeurIPS/ICML/ICLR/ACL.
• Strong understanding of evaluation pitfalls: contamination, overfitting, irreproducibility.
• Engineering ability: Python expertise, sandboxed/distributed execution, CI.
• Hands-on experience evaluating agentic or multi-step LLM systems.

Responsabilidades

• Design and build a unified system for evaluating agentic and algorithmic solutions across workloads, tracking task success, quality and robustness.
• Mine real commits and traces, run sandboxed execution grading, and build task suites reflecting real agentic work.
• Enforce controls against contamination, overfitting, and metric gaming; keep baselines stable and reproducible.
• Compare algorithmic and agentic approaches; determine which to ship based on resolved-task quality.
• Lead external benchmark co-publications and ensure alignment with peer and customer review.
• Incorporate results into product decisions and customer communications.

Descrição da oferta de emprego

Callosum is building a unified benchmarking system for evaluating agentic and algorithmic AI work. The role focuses on reproducible, contamination‑controlled evaluation across motifs and decomposition strategies, grounded in real execution traces and sandboxed grading.

You will architect and implement pipelines that measure task success, quality and robustness, influencing which approaches ship and how we communicate proof points to customers.