Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie

United States

On-site

USD 200,000 - 340,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Snorkel AI is hiring researchers to support the Open Benchmarks Red Team program. You will identify weaknesses in frontier AI benchmarks, reproduce issues, and collaborate with benchmark authors to develop fixes and reusable methods across tasks and datasets.

This hands-on role combines AI evaluation, adversarial testing, and practical benchmark repairs with opportunities to contribute to open-source tools, technical reports, and publications, while advancing benchmark quality.

Qualifications

  • Strong ML evaluation background and Python skills.
  • Experience designing rigorous experiments and analyzing model behavior.
  • Ability to identify weaknesses in evaluation methods and test solutions.
  • Experience driving projects from problem to implementation.

Responsibilities

  • Red-team frontier benchmarks and reproduce issues.
  • Develop benchmark repairs and verify fixes.
  • Develop reusable methods and checks.
  • Collaborate with benchmark authors and publish findings.
  • Contribute to technical reports, tools, publications.

Skills

Python
ML evaluation
Adversarial testing
Experiment design
Research leadership
Communication
Independent drive
Collaboration

Education

PhD in CS/ML or related field
Equivalent research/industry experience

Tools

RL environments
Evaluation frameworks
Open-source contributions

Job description

Snorkel AI is hiring researchers to support the Open Benchmarks Red Team program. You will identify weaknesses in frontier AI benchmarks, reproduce issues, and collaborate with benchmark authors to develop fixes and reusable methods across tasks and datasets.

This hands-on role combines AI evaluation, adversarial testing, and practical benchmark repairs with opportunities to contribute to open-source tools, technical reports, and publications, while advancing benchmark quality.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Red Team AI Benchmark Scientist & Engineer
Red Team AI Benchmark Scientist & Engineer

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Frontier AI Benchmark Research Scientist
Frontier AI Benchmark Research Scientist

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Benchmark Red Team Research Scientist
Benchmark Red Team Research Scientist

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Research Fellow: Open Benchmarks & AI Evaluation
Research Fellow: Open Benchmarks & AI Evaluation

Snorkel AI • San Francisco (CA)

Hybrid
USD 117,000 - 227,000
Research advisor
Compute resources
Open-source artifact
+1
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Remote Genie • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

snorkelai • United States

Remote
USD 120,000 - 180,000
Lead Researcher — Frontier AI Benchmarks & Evaluation
Lead Researcher — Frontier AI Benchmarks & Evaluation

Snorkel AI • United States

Remote
USD 231,000 - 390,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
AI Benchmarks Research Fellow
AI Benchmarks Research Fellow

Aitrainergigs • San Francisco (CA), New York (NY)

Hybrid
USD 117,000 - 227,000
Co-advisor
Compute resources
Open-source artifact
+2