Benchmark Red Team Research Scientist

Snorkel AI

United States

On-site

USD 200,000 - 340,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Snorkel AI in the United States is seeking a Research Scientist / Research Engineer to identify weaknesses in frontier AI benchmarks, help develop fixes, and create reusable methods to improve benchmark quality across tasks and datasets. The role combines hands-on evaluation with collaboration with benchmark authors.

You will design rigorous experiments, analyze model behavior, and contribute to open-source tools and publications, driving impact in benchmark evaluation and improvement.

Qualifications

  • Strong ML/LLM/AI agent evaluation background.
  • Strong Python programming skills with AI agents/evaluation systems.
  • Ability to identify weaknesses in evaluation methods and test rigor.
  • Experience designing rigorous experiments and analyzing model behavior.

Responsibilities

  • Red-team frontier benchmarks and reproduce issues (reward hacking, grader failures, ambiguity, shortcuts).
  • Develop benchmark repairs and verify fixes with benchmark authors.
  • Develop reusable methods and checks to detect recurring weaknesses.
  • Collaborate with benchmark authors and support improved releases.
  • Drive research impact via technical reports, open-source tools, publications.

Skills

Machine learning
LLMs
AI agents
Evaluation
Python
Experiment design
Communication
Collaboration

Education

PhD in Computer Science or Machine Learning

Job description

Snorkel AI in the United States is seeking a Research Scientist / Research Engineer to identify weaknesses in frontier AI benchmarks, help develop fixes, and create reusable methods to improve benchmark quality across tasks and datasets. The role combines hands-on evaluation with collaboration with benchmark authors.

You will design rigorous experiments, analyze model behavior, and contribute to open-source tools and publications, driving impact in benchmark evaluation and improvement.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Red Team AI Benchmark Scientist & Engineer
Red Team AI Benchmark Scientist & Engineer

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Red Team Research Scientist: AI Benchmark Evaluation
Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie • United States

On-site
USD 200,000 - 340,000
Frontier AI Benchmark Research Scientist
Frontier AI Benchmark Research Scientist

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Lead Researcher, Open Benchmarks & Grants
Lead Researcher, Open Benchmarks & Grants

Snorkel AI • United States

Remote
USD 120,000 - 180,000
Senior Research Scientist: Frontier AI Benchmarks & Datasets
Senior Research Scientist: Frontier AI Benchmarks & Datasets

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 180,000 - 260,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Remote Genie • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
AI Benchmarking & Evaluation Scientist
AI Benchmarking & Evaluation Scientist

snorkelai • United States

Remote
USD 120,000 - 180,000
Senior Research Scientist, Frontier AI Data & Benchmarks
Senior Research Scientist, Frontier AI Data & Benchmarks

Snorkel AI • United States

On-site
USD 180,000 - 240,000