Remote Research Scientist, AI Benchmark Evaluation

Ladders

United States

Remote

USD 198,000 - 340,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Cutting-edge research
Funding for AI research
Open-source contributions
Remote work
Collaborative environment

Job summary

Ladders is seeking a Research Scientist for Open Benchmarks Red Team to strengthen AI evaluation by identifying weaknesses in frontier benchmarks and guiding robust releases. The role blends research and engineering, focusing on LLMs, AI agents, and experimental analysis of model behavior.

Work with researchers and engineers to reproduce benchmark issues, develop fixes with authors, and contribute to reports, tools, and publications that influence the AI evaluation ecosystem.

Qualifications

  • Strong background in ML research or engineering focused on evaluation.
  • Proficient in Python with experience in AI agents or evaluation systems.
  • Ability to design experiments and analyze model behavior.
  • Familiarity with LLM benchmarks and evaluation frameworks.
  • Proven track record of independent research and collaboration.

Responsibilities

  • Identify and reproduce issues in frontier benchmarks, such as reward hacking and task ambiguity
  • Propose and implement fixes for validated benchmark issues in collaboration with authors
  • Develop reusable methods to detect recurring benchmark weaknesses
  • Collaborate with benchmark authors to review findings and support improved releases
  • Contribute to technical reports, open-source tools, and publications on benchmark evaluation

Skills

ML research
Python
Evaluation methods
Experiment design
Communication

Job description

Ladders is seeking a Research Scientist for Open Benchmarks Red Team to strengthen AI evaluation by identifying weaknesses in frontier benchmarks and guiding robust releases. The role blends research and engineering, focusing on LLMs, AI agents, and experimental analysis of model behavior.

Work with researchers and engineers to reproduce benchmark issues, develop fixes with authors, and contribute to reports, tools, and publications that influence the AI evaluation ecosystem.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

Ladders • United States

Remote
USD 198,000 - 340,000
Cutting-edge research
Funding for AI research
Open-source contributions
+2
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

snorkelai • United States

Remote
USD 120,000 - 180,000
Red Team Research Scientist: AI Benchmark Evaluation
Red Team Research Scientist: AI Benchmark Evaluation

Remote Genie • United States

On-site
USD 200,000 - 340,000
Benchmark Red Team Research Scientist
Benchmark Red Team Research Scientist

Snorkel AI • United States

On-site
USD 200,000 - 340,000
Remote AI Benchmark & Red-Teaming Specialist
Remote AI Benchmark & Red-Teaming Specialist

AI Work Expert • San Francisco (CA)

Remote
USD 110,000 - 170,000
AI Benchmark & Red-Teaming Research Lead
AI Benchmark & Red-Teaming Research Lead

Turing • United States

Remote
USD 83,000 - 124,000
Red Team AI Benchmark Scientist & Engineer
Red Team AI Benchmark Scientist & Engineer

Snorkel AI • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 340,000
Frontier AI Benchmark Scientist (Red Team)
Frontier AI Benchmark Scientist (Red Team)

Zekai • New York (NY), San Francisco (CA)

Hybrid
USD 200,000 - 375,000
Research Scientist
Research Scientist

Anyone AI Inc. • United States

Remote
USD 110,000 - 160,000
Research Lead, Open Benchmarks Grants
Research Lead, Open Benchmarks Grants

snorkelai • United States

Remote
USD 231,000 - 390,000