Research Scientist, Open Benchmarks Red Team

Ladders

United States

Remote

USD 198,000 - 340,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Cutting-edge research
Funding for AI research
Open-source contributions
Remote work
Collaborative environment

Job summary

Ladders is seeking a Research Scientist for Open Benchmarks Red Team to strengthen AI evaluation by identifying weaknesses in frontier benchmarks and guiding robust releases. The role blends research and engineering, focusing on LLMs, AI agents, and experimental analysis of model behavior.

Work with researchers and engineers to reproduce benchmark issues, develop fixes with authors, and contribute to reports, tools, and publications that influence the AI evaluation ecosystem.

Qualifications

  • Strong background in ML research or engineering focused on evaluation.
  • Proficient in Python with experience in AI agents or evaluation systems.
  • Ability to design experiments and analyze model behavior.
  • Familiarity with LLM benchmarks and evaluation frameworks.
  • Proven track record of independent research and collaboration.

Responsibilities

  • Identify and reproduce issues in frontier benchmarks, such as reward hacking and task ambiguity
  • Propose and implement fixes for validated benchmark issues in collaboration with authors
  • Develop reusable methods to detect recurring benchmark weaknesses
  • Collaborate with benchmark authors to review findings and support improved releases
  • Contribute to technical reports, open-source tools, and publications on benchmark evaluation

Skills

ML research
Python
Evaluation methods
Experiment design
Communication

Job description

For our client, we are seeking a Research Scientist, Open Benchmarks Red Team to join the team of a leader in the Information Technology space.

This role focuses on strengthening the reliability and rigor of AI evaluation by finding weaknesses in frontier benchmarks and helping improve how they are assessed. You will work at the intersection of research and engineering, developing practical methods to identify issues, validate findings, and support better benchmark releases. The position calls for close collaboration with researchers and engineers while contributing to technical reports, tools, and publications that influence the broader AI evaluation ecosystem.

The work is highly technical and research-driven, with a strong emphasis on LLMs, AI agents, and experimental analysis of model behavior.

Location: Remote

Compensation: $198,000 – $340,000 annually

Responsibilities
  • Identify and reproduce issues in frontier benchmarks, such as reward hacking and task ambiguity
  • Propose and implement fixes for validated benchmark issues in collaboration with authors
  • Develop reusable methods to detect recurring benchmark weaknesses
  • Collaborate with benchmark authors to review findings and support improved releases
  • Contribute to technical reports, open-source tools, and publications on benchmark evaluation
Qualifications
  • Strong research or engineering background in machine learning, LLMs, AI agents, and evaluation
  • Proficient in Python programming with experience in AI agents or evaluation systems
  • Ability to identify weaknesses in evaluation methods and develop new approaches
  • Experience in designing experiments and analyzing model behavior
  • Familiarity with LLM/agent benchmarks and evaluation frameworks
  • Proven track record of independently driving research and engineering projects
  • Strong communication and collaboration skills
Benefits
  • Opportunity to work on cutting-edge AI evaluation projects
  • Collaborative environment with leading researchers and engineers
  • Access to significant funding support for innovative AI research
  • Support for professional development through publications and open-source contributions
  • Engagement in impactful research that shapes the future of AI benchmarks

Our client is an equal opportunity employer.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

snorkelai • United States

Remote
USD 120,000 - 180,000
Remote Research Scientist, AI Benchmark Evaluation
Remote Research Scientist, AI Benchmark Evaluation

Ladders • United States

Remote
USD 198,000 - 340,000
Cutting-edge research
Funding for AI research
Open-source contributions
+2
Staff Research Scientist
Staff Research Scientist

turing • United States

Remote
USD 250,000 - 400,000
Research Lead, Open Benchmarks Grants
Research Lead, Open Benchmarks Grants

snorkelai • United States

Remote
USD 231,000 - 390,000
Remote AI Benchmark Researcher - Computational STEM
Remote AI Benchmark Researcher - Computational STEM

Weekday • United States

Remote
USD 83,000 - 124,000
AI Benchmark & Red-Teaming Specialist
AI Benchmark & Red-Teaming Specialist

Turing • United States

Remote
USD 83,000 - 124,000
Member of Technical Staff, Realm Lab (frontier AI research)
Member of Technical Staff, Realm Lab (frontier AI research)

Confidential • United States

Remote
USD 240,000 - 350,000
Equity compensation
Performance bonuses
Health insurance reimbursement
+2
AI Research Engineer
AI Research Engineer

Cerebro • San Francisco (CA)

On-site
USD 270,000 - 330,000
Equity
AI Benchmark & Red-Teaming Specialist
AI Benchmark & Red-Teaming Specialist

HumanitApp • San Francisco (CA)

On-site
USD 83,000 - 138,000
Research Marketing Lead - Remote (Worldwide)
Research Marketing Lead - Remote (Worldwide)

RemoteJobsOne • United States

Remote
USD 250,000 - 350,000
Equity compensation
Performance bonuses
Health insurance reimbursement
+3