Staff Evaluations Engineer — AI Benchmarks

Ai

Greater London

On-site

GBP 85,000 - 130,000

Full time

11 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

30 days annual leave
Pension plan with 10% employer contrib
Bupa private health cover
London or SF based role

Job summary

Ai is a stealth startup building infrastructure for AI and biology data generation. You will design and own how we evaluate models and agents, turning scientific workflows into benchmarks, checks and reproducible tests that work at scale.

Based in London or San Francisco, you will partner with researchers to trace results to their causes, protect against leakage, and report uncertainty. This role reports into the intelligence team and includes generous leave and private health benefits.

Qualifications

  • Experience building evaluations, benchmarks or task suites for language models or agents.
  • Experience with long-horizon tasks using tools like search, databases or code execution.
  • Ability to detect reward hacking and data leakage that misleads scores.
  • Understanding statistics for noisy results: repeats and uncertainty.
  • Experience calibrating judges against expert ratings.
  • Hands-on with evaluation frameworks and reproducible pipelines.
  • Clear writing and communication about evaluation design and limitations.

Responsibilities

  • Define benchmarks and evaluation protocols for models and agents.
  • Turn scientists' workflows into test suites and checks.
  • Benchmark results with uncertainty and traceability.
  • Own data used for training and evaluation; prepare datasets.

Skills

Python
Benchmarking
Long-horizon tasks
Evaluation design
Reward hacking detection
Statistics
Clear writing

Tools

Harbor
Inspect
Prime Intellect verifiers

Job description

Ai is a stealth startup building infrastructure for AI and biology data generation. You will design and own how we evaluate models and agents, turning scientific workflows into benchmarks, checks and reproducible tests that work at scale.

Based in London or San Francisco, you will partner with researchers to trace results to their causes, protect against leakage, and report uncertainty. This role reports into the intelligence team and includes generous leave and private health benefits.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Staff AI Benchmark Engineer
Staff AI Benchmark Engineer

Stealth - AI x Bio • Greater London

On-site
GBP 90,000 - 150,000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Stealth - AI x Bio • Greater London

On-site
GBP 90,000 - 150,000
AI Evaluation Engineer: Agentic Benchmarks & Production
AI Evaluation Engineer: Agentic Benchmarks & Production

Teli Labs • Greater London

On-site
GBP 90,000 - 130,000
AI/ML Data Scientist — Hybrid London, Own AI Benchmarks
AI/ML Data Scientist — Hybrid London, Own AI Benchmarks

Medly AI • Greater London

Hybrid
GBP 65,000 - 90,000
AI Evaluation Engineer — Hybrid, Public-Impact Tech
AI Evaluation Engineer — Hybrid, Public-Impact Tech

i.AI • City Of London

Hybrid
GBP 46,000 - 63,000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Ai • Greater London

On-site
GBP 85,000 - 130,000
30 days annual leave
Pension plan with 10% employer contrib
Bupa private health cover
+1
Agentic Evaluation Engineer — Benchmarks & Systems
Agentic Evaluation Engineer — Benchmarks & Systems

AI Startups UK • Greater London

Hybrid
GBP 90,000 - 130,000
Competitive salary
Equity ownership
Private healthcare
+1
Staff Research Engineer: Agentic Evaluation & Benchmarks
Staff Research Engineer: Agentic Evaluation & Benchmarks

Callosum Technologies Ltd. • Greater London

Hybrid
GBP 90,000 - 140,000
Competitive salary
Equity & Ownership
Private healthcare
+2
Senior ML QA Engineer: Benchmark & Validate AI Systems
Senior ML QA Engineer: Benchmark & Validate AI Systems

EngineersOfAI • Greater London

On-site
GBP 60,000 - 80,000
Unlimited annual leave
Up to 5% matched pension
Phantom equity
+2
Senior AI Evaluation Engineer: Public Sector Impact, Hybrid
Senior AI Evaluation Engineer: Public Sector Impact, Hybrid

i.AI • City Of London

Hybrid
GBP 58,000 - 87,000
Hybrid working
Annual leave and learning stipend
Parental leave
+1