Get a reply from this employer
Your resume and cover letter, tailored to exactly what they’re hiring for.
Openbenchmarks is seeking a rigorous evaluator to turn raw benchmark outputs into reports that withstand expert scrutiny. You will ground conclusions in solid statistics, design evaluations, and clear technical writing, and you will craft honest, reproducible charts in code.
You will identify the real findings in messy results, explain failures rather than conceal them, and communicate results to technical audiences. This role offers the chance to shape how AI tools are compared and interpreted.
The role centers on turning raw benchmark output into reports that hold up to expert scrutiny. A strong candidate brings solid statistics, hands on evaluation design, and clear technical writing, plus the ability to build honest charts in code. The work also involves finding the real finding in messy results and explaining failures rather than hiding them.
AI makes it cheap to generate comparisons and claims, so the human job is to make sure each number is controlled, tested and reproducible. The writing and judgment about what a result actually means stay central, since agents and people both lean on these evaluations to choose tools. Careful statistics and clear graphs become the trust layer that AI driven decisions rest on.
The role builds direct experience designing and running evaluations of AI tools and agents, including grading rules and judge setup. It also grows skill in spotting where evaluations fail and in communicating results to technical audiences. Working alongside researchers on real benchmark data is the practical path to that experience.
This is our own read, written from the employer's posting and the company's Spotlight. The employer's posting is the source of truth for the role.
Openbenchmarks runs independent, open-source benchmarks that compare APIs, tools and agents on the same tasks under the same conditions, per the company.