Research Scientist, Open Benchmarks Red Team

snorkelai

United States

Remote

USD 120,000 - 180,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

snorkelai is seeking a researcher/engineer to help make evaluations of advanced AI systems more reliable by identifying weaknesses in benchmarks and collaborating with authors to fix them. You will combine hands-on testing of models and agents with research into recurring evaluation failures and engineering improvements that can be reused across tasks.

Requirements include ML/AI evaluation experience, strong Python skills for building and debugging evaluation systems, and the ability to quantify

Qualifications

  • Experience in ML, LLMs, AI agents, and evaluation.
  • Strong Python skills for building, running, and debugging evaluation systems.
  • Ability to identify limitations in evaluation methods, develop and test alternatives.
  • Experience designing experiments, analyzing model behavior, and investigating failures.
  • Familiarity with benchmarks, evaluation frameworks, or automated graders.
  • Ability to take research or engineering work from discovery through implementation and validation, and communicate findings clearly.

Responsibilities

  • Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated.
  • Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions.
  • Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets.
  • Share findings with authors, coordinate improvements, and help prepare updated benchmark releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.

Skills

ML research
Python programming
Evaluation methods
Agent evaluation

Job description

Role overview

Help make evaluations of advanced AI systems more reliable by finding weaknesses in benchmarks and working with their authors to validate and fix them. The work combines hands-on testing of models and agents, research into recurring evaluation failures, and engineering improvements that can be reused across tasks.

Responsibilities
  • Probe benchmarks for grading errors, ambiguous tasks, exploitable shortcuts, reward hacking, and environment vulnerabilities; reproduce issues so they can be investigated.
  • Propose repairs for confirmed problems, implement selected changes with benchmark authors, and check that fixes preserve valid solutions.
  • Build or adapt tools and methods that can detect recurring benchmark weaknesses across tasks and datasets.
  • Share findings with authors, coordinate improvements, and help prepare updated benchmark releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to evaluating benchmarks.
Requirements
  • Research or engineering experience in machine learning, large language models, AI agents, and evaluation.
  • Strong Python skills and experience building, running, and debugging agents or evaluation systems.
  • Ability to identify limitations in evaluation methods, develop alternatives, and test them rigorously.
  • Experience designing experiments, analyzing model behavior, and investigating unexpected failures.
  • Familiarity with agent or language-model benchmarks, evaluation frameworks, or automated graders.
  • Ability to take research or engineering work from problem discovery through implementation and validation, and communicate findings clearly.
Nice to have
  • Experience with adversarial testing, red teaming, reward hacking, or specification gaming.
  • Work on benchmarks for coding agents, computer use, or professional workflows; familiarity with reinforcement-learning environments, agent trajectories, verifiers, or evaluation infrastructure.
  • Publications or open-source work in AI evaluation, robustness, or agent systems. A relevant doctorate or equivalent research or industry experience is also valued.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Research Lead, Open Benchmarks Grants
Research Lead, Open Benchmarks Grants

snorkelai • United States

Remote
USD 231,000 - 390,000
Red Team Research Scientist: AI Benchmark Evaluation
Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie • United States

On-site
USD 200,000 - 340,000
Research Scientist
Research Scientist

Anyone AI Labs • United States

Remote
USD 120,000 - 180,000
Research Scientist, Agent Evaluation
Research Scientist, Agent Evaluation

Reinforced Learning • Town of Tonawanda (NY)

On-site
USD 120,000 - 170,000
Staff Research Scientist
Staff Research Scientist

turing • United States

Remote
USD 250,000 - 400,000
AI Benchmark & Red-Teaming Specialist
AI Benchmark & Red-Teaming Specialist

Turing • United States

Remote
USD 83,000 - 124,000
Remote AI Benchmark & Red-Teaming Specialist
Remote AI Benchmark & Red-Teaming Specialist

AI Work Expert • San Francisco (CA)

Remote
USD 110,000 - 170,000
Red Team AI Benchmark Scientist & Engineer
Red Team AI Benchmark Scientist & Engineer

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Benchmark Red Team Research Scientist
Benchmark Red Team Research Scientist

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Bake AI Inc. • San Mateo (CA)

On-site
USD 180,000 - 240,000