AI Benchmark Engineer: Reproducible Evaluation System

Callosum; Greater London; Presencial; GBP 90.000 - 150.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Competitive salary

Equity ownership

Private healthcare

Visa sponsorship and relocation

London office – in person

Descripción de la vacante

Callosum, based in London, is building the benchmarking system for evaluating agentic and algorithmic AI across tasks. You will design and implement a unified evaluation pipeline and ensure reproducible results that inform product decisions.

This role requires a PhD or strong research track record, excellent Python, and hands-on experience with sandboxed and distributed execution. London-based, in-person work with visa sponsorship possible.

Formación

• PhD in computer science, ML, or related field.
• Authorship or co-authorship of a benchmark or evaluation paper.
• Strong understanding of how evaluation goes wrong in LLM contexts.
• Excellent Python programming and CI/distro execution experience.
• Hands-on experience building or rigorously evaluating agentic or multi-step LLM systems.

Responsabilidades

• Design and build a unified benchmarking system for task success, quality, and robustness.
• Mine real commits and traces, run sandboxed grading, and assemble task suites.
• Publish or co-publish benchmark results and contribute to external publications.
• Decide which approach ships based on evidence and quality claims.
• Lead external benchmark collaborations and ensure reproducibility.
• Collaborate with cross-functional teams to integrate benchmarking into product decisions.

Conocimientos

Python

Sandboxed execution

Distributed execution

Agentic ML

ML benchmarks

Open-source tooling

Educación

PhD in computer science / ML or related field

Herramientas

CI pipelines

Sandboxed execution environments

Descripción del empleo

Callosum, based in London, is building the benchmarking system for evaluating agentic and algorithmic AI across tasks. You will design and implement a unified evaluation pipeline and ensure reproducible results that inform product decisions.

This role requires a PhD or strong research track record, excellent Python, and hands-on experience with sandboxed and distributed execution. London-based, in-person work with visa sponsorship possible.