Red Team AI Benchmark Scientist & Engineer

Snorkel AI

New York, San Francisco (NY, CA)

Hybrid

USD 200,000 - 340,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Snorkel AI is hiring researchers to support the Open Benchmarks Red Team program. You will identify benchmark weaknesses, help develop and validate fixes, and create reusable methods to improve benchmark quality across tasks and datasets.

This hands-on role combines AI evaluation, adversarial testing, practical benchmark repairs, and collaboration with benchmark authors to drive research impact and contribute to reports and open-source tools.

Qualifications

  • Strong Python programming skills and experience building, running, and debugging AI agents or evaluation systems.
  • Experience with ML/LLM evaluation, benchmarks, or automated grading systems.
  • Ability to design rigorous experiments and analyze model behavior.

Responsibilities

  • Red-team frontier benchmarks: identify and reproduce issues like reward hacking and grader failures.
  • Develop benchmark repairs and verify fixes with benchmark authors without breaking valid solutions.
  • Create reusable methods and checks to detect recurring benchmark weaknesses.
  • Collaborate with benchmark authors and support official improved releases.
  • Contribute to technical reports, open-source tools, publications, and new approaches to benchmark evaluation.

Skills

Python programming
ML/AI evaluation
Experiment design
Benchmark evaluation
Independent project execution
Communication & collaboration

Job description

Snorkel AI is hiring researchers to support the Open Benchmarks Red Team program. You will identify benchmark weaknesses, help develop and validate fixes, and create reusable methods to improve benchmark quality across tasks and datasets.

This hands-on role combines AI evaluation, adversarial testing, practical benchmark repairs, and collaboration with benchmark authors to drive research impact and contribute to reports and open-source tools.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Red Team Research Scientist: AI Benchmark Evaluation
Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie • United States

On-site
USD 200,000 - 340,000
Frontier AI Benchmark Research Scientist
Frontier AI Benchmark Research Scientist

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Benchmark Red Team Research Scientist
Benchmark Red Team Research Scientist

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Remote Genie • United States

On-site
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
AI Benchmarks Research Fellow
AI Benchmarks Research Fellow

Aitrainergigs • San Francisco (CA), New York (NY)

Hybrid
USD 117,000 - 227,000
Co-advisor
Compute resources
Open-source artifact
+2
Research Fellow: Open Benchmarks & AI Evaluation
Research Fellow: Open Benchmarks & AI Evaluation

Snorkel AI • San Francisco (CA)

Hybrid
USD 117,000 - 227,000
Research advisor
Compute resources
Open-source artifact
+1
Research Fellow, Open AI Benchmarks & Datasets
Research Fellow, Open AI Benchmarks & Datasets

Snorkel AI • United States

Hybrid
USD 117,000 - 227,000
Research co-advisor
Compute support
Open-source artifact
+2