Staff AI Benchmark Engineer

Stealth - AI x Bio; Greater London; In loco; GBP 90.000 - 150.000; Tempo pieno

Descrizione del lavoro

Stealth - AI x Bio is seeking a role focused on designing and maintaining benchmarks to evaluate language models, agents, and scientific models. You will own how evaluations are built, validated, and interpreted alongside scientists and engineers in a small, elite team.

You will create benchmarks that remain robust as models evolve, guard against data leakage and score manipulation, and work with data pipelines to ensure reproducibility and transparency in reporting results.

Competenze

• Building evaluations, benchmarks or task suites for language models or agents, including the scoring code.
• Long-horizon agent tasks that use tools such as search, databases and code execution.
• Detecting reward hacking, grader exploits, data contamination and label leakage, and the other ways a score can mislead.
• Statistics for noisy results: repeats, uncertainty and deciding when two runs really differ.
• Model-based grading and rubric design, including calibrating LLM judges against expert ratings.
• Hands-on experience with evaluation frameworks such as Harbor, Inspect or Prime Intellect's verifiers, and building evaluations that rerun reproducibly.
• Clear writing and communication about evaluation design, results and their limitations.
• Strong Python and daily use of coding agents.

Mansioni

• Own how we evaluate our models and agents, including benchmarks and checks.
• Turn scientists' workflows into benchmarks that remain hard as models improve.
• Protect benchmarks against leakage, memorization and gaming, and report results with uncertainty.
• Build data evaluation pipelines, sourcing external data and preparing experimental datasets.
• Collaborate with intelligence, RL, Bio ML and infrastructure teams to drive evaluation strategy.

Conoscenze

Evaluations benchmarks

Python programming

Long-horizon tasks

Statistics

Model-based grading

Communication

Strumenti

Harbor

Inspect

Prime Intellect verifiers

vLLM

Descrizione del lavoro

Stealth - AI x Bio is seeking a role focused on designing and maintaining benchmarks to evaluate language models, agents, and scientific models. You will own how evaluations are built, validated, and interpreted alongside scientists and engineers in a small, elite team.

You will create benchmarks that remain robust as models evolve, guard against data leakage and score manipulation, and work with data pipelines to ensure reproducibility and transparency in reporting results.