Member of Technical Staff, Evaluations

Stealth - AI x Bio; Greater London; Presencial; GBP 90.000 - 150.000; Jornada completa

Descripción de la vacante

Stealth - AI x Bio is seeking a role focused on designing and maintaining benchmarks to evaluate language models, agents, and scientific models. You will own how evaluations are built, validated, and interpreted alongside scientists and engineers in a small, elite team.

You will create benchmarks that remain robust as models evolve, guard against data leakage and score manipulation, and work with data pipelines to ensure reproducibility and transparency in reporting results.

Formación

• Building evaluations, benchmarks or task suites for language models or agents, including the scoring code.
• Long-horizon agent tasks that use tools such as search, databases and code execution.
• Detecting reward hacking, grader exploits, data contamination and label leakage, and the other ways a score can mislead.
• Statistics for noisy results: repeats, uncertainty and deciding when two runs really differ.
• Model-based grading and rubric design, including calibrating LLM judges against expert ratings.
• Hands-on experience with evaluation frameworks such as Harbor, Inspect or Prime Intellect's verifiers, and building evaluations that rerun reproducibly.
• Clear writing and communication about evaluation design, results and their limitations.
• Strong Python and daily use of coding agents.

Responsabilidades

• Own how we evaluate our models and agents, including benchmarks and checks.
• Turn scientists' workflows into benchmarks that remain hard as models improve.
• Protect benchmarks against leakage, memorization and gaming, and report results with uncertainty.
• Build data evaluation pipelines, sourcing external data and preparing experimental datasets.
• Collaborate with intelligence, RL, Bio ML and infrastructure teams to drive evaluation strategy.

Conocimientos

Evaluations benchmarks

Python programming

Long-horizon tasks

Statistics

Model-based grading

Communication

Herramientas

Harbor

Inspect

Prime Intellect verifiers

vLLM

Descripción del empleo

The opportunity

We're a stealth startup in AI and bio, building infrastructure for data generation. The team is small and elite, the problems are hard, and the foundations are being laid right now. You'd help write them from the first line, with real ownership and a direct line to the founders.

About us

AI for biology has a data problem. The data that matters most doesn't exist yet, so we're building the infrastructure to produce it, with quality and traceability built in from day one. We're in stealth and heads-down on execution. We'll share more about what we're building once you've spoken with the team.

The role

You will own how we evaluate our models and agents. That covers the biological models the agents use as tools as well as the agents themselves, and both how agents plan scientific work and how they carry it out. You will work with our scientists to turn their workflows into benchmarks for long-horizon scientific tasks, write the checks that decide whether an answer is correct and supported by the data, and run the internal benchmark we use to compare models and harnesses. You will design benchmarks that stay hard as models improve, protect them against leakage, memorised answers and gaming, and report results with the uncertainty they carry. When results shift unexpectedly, you will trace the change to its cause, whether that is the model, the harness or the evaluation itself.

You will also own the data the team trains and evaluates on: sourcing external datasets, preparing our own experimental data, and building the tools researchers use to understand where models fail. Scientific data is noisy and grows over time, so every evaluation must account for noise, be easy to rerun, and be documented well enough for others to trust.

You will report into the intelligence team and work closely with the RL, Bio ML and infrastructure engineers, as well as with our scientists.

Who you are

You have built evaluations or benchmarks for language models, agents or scientific models, and you have found problems in results that others had taken at face value. You read the traces before trusting a score, and treat an unexpectedly high result as a bug until it has been checked. A background in biology is not required, as you will be working closely with our scientists.

MUST HAVE

• Building evaluations, benchmarks or task suites for language models or agents, including the scoring code.
• Long-horizon agent tasks that use tools such as search, databases and code execution.
• Detecting reward hacking, grader exploits, data contamination and label leakage, and the other ways a score can mislead.
• Statistics for noisy results: repeats, uncertainty and deciding when two runs really differ.
• Model-based grading and rubric design, including calibrating LLM judges against expert ratings.
• Hands-on experience with evaluation frameworks such as Harbor, Inspect or Prime Intellect's verifiers, and building evaluations that rerun reproducibly.
• Clear writing and communication about evaluation design, results and their limitations.
• Strong Python and daily use of coding agents.

NICE TO HAVE

• Experience evaluating models on biological or other experimental data.
• Public benchmarks or evaluation libraries you built or contributed to.
• Familiarity with how agentic benchmarks such as SWE-bench Verified or Terminal-Bench are built, and the ways they can be exploited.
• Running agent evaluations in containers or sandboxes, in parallel and at volume.
• Inference engines such as vLLM or SGLang, for serving the models under evaluation.

How we work

The role is based in London or San Francisco. We work in the open, write decisions down, and expect each person to own their area and make the calls inside it.

Everyone gets 30 days of annual leave plus local public holidays. In the UK that comes with a pension with a 10% employer contribution and Bupa private health cover; in the US, a 401(k) with matching and health cover. We add to these as the team grows.

Our process

Four stages: a screening call, a behavioural interview, an in-person technical session with the team on a real problem from our work, and references.

We are an equal opportunity employer. We make hiring decisions on merit, scope-fit, and the strength of the working relationship we expect to build with each hire. Applications welcome from candidates of any background.