Frontier AI Benchmark Scientist (Red Team)

Zekai

New York, San Francisco (NY, CA)

Hybrid

USD 200,000 - 375,000

Full time

6 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Snorkel AI is hiring researchers to support the Open Benchmarks Red Team program and improve frontier AI evaluations. This hands‑on role combines AI evaluation, adversarial testing, and practical benchmark repairs with collaboration across benchmark authors and teams.

You will identify weaknesses, develop fixes, and create reusable methods to raise benchmark quality across tasks and datasets, contributing to reports and open‑source tools.

Qualifications

  • Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation.
  • Strong Python programming skills and experience building, running, and debugging AI agents or evaluation systems.
  • Ability to identify weaknesses in existing evaluation methods, develop new approaches, and rigorously test whether they work.

Responsibilities

  • Red‑team frontier benchmarks: Identify and reproduce issues such as reward hacking, grader failures, task ambiguity, shortcuts, and environment vulnerabilities.
  • Develop benchmark repairs: Propose fixes, implement with benchmark authors, and verify not to break valid solutions.
  • Develop reusable methods: Create checks to detect recurring benchmark weaknesses and reuse existing tools where possible.
  • Collaborate with benchmark authors: Review findings before publication and support improved releases.
  • Drive research impact: Contribute to technical reports, open‑source tools, publications, and new evaluation approaches.

Skills

Python
Research
LLMs
Evaluation
ML

Job description

Snorkel AI is hiring researchers to support the Open Benchmarks Red Team program and improve frontier AI evaluations. This hands‑on role combines AI evaluation, adversarial testing, and practical benchmark repairs with collaboration across benchmark authors and teams.

You will identify weaknesses, develop fixes, and create reusable methods to raise benchmark quality across tasks and datasets, contributing to reports and open‑source tools.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Frontier AI Benchmark Research Scientist
Frontier AI Benchmark Research Scientist

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Red Team Research Scientist: AI Benchmark Evaluation
Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie • United States

On-site
USD 200,000 - 340,000
Red Team AI Benchmark Scientist & Engineer
Red Team AI Benchmark Scientist & Engineer

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Benchmark Red Team Research Scientist
Benchmark Red Team Research Scientist

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Senior Research Scientist: Frontier AI Benchmarks & Datasets
Senior Research Scientist: Frontier AI Benchmarks & Datasets

AItoolnavio • New York (NY), San Francisco (CA)

Hybrid
USD 180,000 - 260,000
Senior Research Scientist, Frontier AI Data & Benchmarks
Senior Research Scientist, Frontier AI Data & Benchmarks

Snorkel AI • United States

On-site
USD 180,000 - 240,000
Lead Researcher — Frontier AI Benchmarks & Evaluation
Lead Researcher — Frontier AI Benchmarks & Evaluation

Snorkel AI • United States

Remote
USD 231,000 - 390,000
Senior Research Scientist - Frontier AI Data & Benchmarks
Senior Research Scientist - Frontier AI Data & Benchmarks

Snorkel AI • San Francisco (CA)

On-site
USD 180,000 - 260,000
Senior Research Scientist, Frontier AI Data & Benchmarks
Senior Research Scientist, Frontier AI Data & Benchmarks

Snorkel AI • New York (NY)

On-site
USD 150,000 - 210,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Remote Genie • United States

On-site
USD 200,000 - 340,000