Benchmark Researcher: AI Evaluation & Critiques (Remote)

Remote; San Francisco (CA); Remote; USD 100,000 - 200,000; Full time

Benefits offered by this job

Health insurance

Life insurance

Pension plan

Paid time off

Parental leave

Equipment budget for tools & training

Paid travel for conferences

Job summary

Epoch AI is seeking a Researcher to develop and publish critiques and reviews of AI benchmarks. You will analyze a wide range of new benchmarks, assess their methodologies, and present findings in public-facing research. You should be comfortable using coding agents to assist, while retaining your own critical judgment.

Examples include reviews of SWE-bench Verified, OSWorld, and economic value benchmarks. The role is fully remote with international hiring, and invites applicants from diverse

Qualifications

• Strong writing skills suitable for publication.
• Ability to evaluate research methodologies critically.
• Experience with AI benchmarks and related evaluations.
• Comfort with analyzing complex benchmark data.

Responsibilities

• Review benchmarks: assess a new benchmark at least every two weeks, evaluating methodologies and performance implications.
• Publish and maintain public-facing reports on new benchmarks and versions.
• Dig into benchmark data details using coding agents while exercising independent judgment.

Skills

Exceptional written communication

Critical thinking about methodologies

Familiar with AI benchmarks

Ability to work with dense data

Tools

Coding agents

Job description

Epoch AI is seeking a Researcher to develop and publish critiques and reviews of AI benchmarks. You will analyze a wide range of new benchmarks, assess their methodologies, and present findings in public-facing research. You should be comfortable using coding agents to assist, while retaining your own critical judgment.

Examples include reviews of SWE-bench Verified, OSWorld, and economic value benchmarks. The role is fully remote with international hiring, and invites applicants from diverse