Role overview
Help make evaluations of advanced AI systems more reliable by finding weaknesses in benchmarks and working with their authors to validate and fix them. The work combines hands-on testing of models and agents, research into recurring evaluation failures, and engineering improvements that can be reused across tasks.
Responsibilities
- Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated.
- Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions.
- Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets.
- Share findings with authors, coordinate improvements, and help prepare updated benchmark releases.
- Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.
Requirements
- Research or engineering experience in machine learning, large language models, AI agents, and evaluation.
- Strong Python skills and experience building, running, and debugging agents or evaluation systems.
- Ability to identify limitations in evaluation methods, develop alternatives, and test them rigorously.
- Experience designing experiments, analyzing model behavior, and investigating unexpected failures.
- Familiarity with agent or language-model benchmarks, evaluation frameworks, or automated graders.
- Ability to take research or engineering work from problem discovery through implementation and validation, and communicate findings clearly.
Nice to have
- Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
- Work on benchmarks for coding agents, computer use, or professional workflows; familiarity with reinforcement-learning environments, agent trajectories, verifiers, or evaluation infrastructure.
- Publications or open-source work in AI evaluation, robustness, or agent systems. A relevant doctorate or equivalent research or industry experience is also valued.