Member of Technical Staff, Evaluations

Ai; Greater London; On-site; GBP 85,000 - 130,000; Full time

Benefits offered by this job

30 days annual leave

Pension plan with 10% employer contrib

Bupa private health cover

London or SF based role

Job summary

Ai is a stealth startup building infrastructure for AI and biology data generation. You will design and own how we evaluate models and agents, turning scientific workflows into benchmarks, checks and reproducible tests that work at scale.

Based in London or San Francisco, you will partner with researchers to trace results to their causes, protect against leakage, and report uncertainty. This role reports into the intelligence team and includes generous leave and private health benefits.

Qualifications

• Experience building evaluations, benchmarks or task suites for language models or agents.
• Experience with long-horizon tasks using tools like search, databases or code execution.
• Ability to detect reward hacking and data leakage that misleads scores.
• Understanding statistics for noisy results: repeats and uncertainty.
• Experience calibrating judges against expert ratings.
• Hands-on with evaluation frameworks and reproducible pipelines.
• Clear writing and communication about evaluation design and limitations.

Responsibilities

• Define benchmarks and evaluation protocols for models and agents.
• Turn scientists' workflows into test suites and checks.
• Benchmark results with uncertainty and traceability.
• Own data used for training and evaluation; prepare datasets.

Skills

Python

Benchmarking

Long-horizon tasks

Evaluation design

Reward hacking detection

Statistics

Clear writing

Tools

Harbor

Inspect

Prime Intellect verifiers

Job description

THE OPPORTUNITY

We're a stealth startup in AI and bio, building infrastructure for data generation. The team is small and elite, the problems are hard, and the foundations are being laid right now. You'd help write them from the first line, with real ownership and a direct line to the founders.

ABOUT US

AI for biology has a data problem. The data that matters most doesn't exist yet, so we're building the infrastructure to produce it, with quality and traceability built in from day one. We're in stealth and heads-down on execution. We'll share more about what we're building once you've spoken with the team.

THE ROLE

You will own how we evaluate our models and agents. That covers the biological models the agents use as tools as well as the agents themselves, and both how agents plan scientific work and how they carry it out. You will work with our scientists to turn their workflows into benchmarks for long-horizon scientific tasks, write the checks that decide whether an answer is correct and supported by the data, and run the internal benchmark we use to compare models and harnesses. You will design benchmarks that stay hard as models improve, protect them against leakage, memorised answers and gaming, and report results with the uncertainty they carry. When results shift unexpectedly, you will trace the change to its cause, whether that is the model, the harness or the evaluation itself. You will also own the data the team trains and evaluates on: sourcing external datasets, preparing our own experimental data, and building the tools researchers use to understand where models fail. Scientific data is noisy and grows over time, so every evaluation must account for noise, be easy to rerun, and be documented well enough for others to trust. You will report into the intelligence team and work closely with the RL, Bio ML and infrastructure engineers, as well as with our scientists.

WHO YOU ARE

You have built evaluations or benchmarks for language models, agents or scientific models, and you have found problems in results that others had taken at face value. You read the traces before trusting a score, and treat an unexpectedly high result as a bug until it has been checked. A background in biology is not required, as you will be working closely with our scientists.

MUST HAVE

• Building evaluations, benchmarks or task suites for language models or agents, including the scoring code.
• Long-horizon agent tasks that use tools such as search, databases and code execution.
• Detecting reward hacking, grader exploits, data contamination and label leakage, and the other ways a score can mislead.
• Statistics for noisy results: repeats, uncertainty and deciding when two runs really differ.
• Model-based grading and rubric design, including calibrating LLM judges against expert ratings.
• Hands‑on experience with evaluation frameworks such as Harbor, Inspect or Prime Intellect's verifiers, and building evaluations that rerun reproducibly.
• Clear writing and communication about evaluation design, results and their limitations.
• Strong Python and daily use of coding agents.

NICE TO HAVE

• Experience evaluating models on biological or other experimental data.
• Public benchmarks or evaluation libraries you built or contributed to.
• Familiarity with how agentic benchmarks such as SWE-bench Verified or Terminal-Bench are built, and the ways they can be exploited.
• Running agent evaluations in containers or sandboxes, in parallel and at volume.
• Inference engines such as vLLM or SGLang, for serving the models under evaluation.

HOW WE WORK

The role is based in London or San Francisco. We work in the open, write decisions down, and expect each person to own their area and make the calls inside it. Everyone gets 30 days of annual leave plus local public holidays. In the UK that comes with a pension with a 10% employer contribution and Bupa private health cover; in the US, a 401(k) with matching and health cover. We add to these as the team grows.

• Everyone gets 30 days of annual leave plus local public holidays.
• In the UK that comes with a pension with a 10% employer contribution and Bupa private health cover; in the US, a 401(k) with matching and health cover.

OUR PROCESS

Four stages: a screening call, a behavioural interview, an in-person technical session with the team on a real problem from our work, and references. We are an equal opportunity employer. We make hiring decisions on merit, scope-fit, and the strength of the working relationship we expect to build with each hire. Applications welcome from candidates of any background.