Benchmarks Storyteller

present of AI

United States

On-site

USD 90,000 - 150,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Openbenchmarks is seeking a rigorous evaluator to turn raw benchmark outputs into reports that withstand expert scrutiny. You will ground conclusions in solid statistics, design evaluations, and clear technical writing, and you will craft honest, reproducible charts in code.

You will identify the real findings in messy results, explain failures rather than conceal them, and communicate results to technical audiences. This role offers the chance to shape how AI tools are compared and interpreted.

Qualifications

  • Solid statistics background demonstrated in benchmarks.
  • Experience designing and running evaluations for AI tools.
  • Ability to communicate results with technical clarity.

Responsibilities

  • Turn raw benchmark outputs into rigorous reports.
  • Ensure results are controlled, tested and reproducible.
  • Identify real findings in messy results and explain failures.

Skills

Statistics
Evaluation design
Technical writing
Data visualization in code
Reproducible charts
Mixed effects modeling

Job description

The role centers on turning raw benchmark output into reports that hold up to expert scrutiny. A strong candidate brings solid statistics, hands on evaluation design, and clear technical writing, plus the ability to build honest charts in code. The work also involves finding the real finding in messy results and explaining failures rather than hiding them.

How it fits the AI era

AI makes it cheap to generate comparisons and claims, so the human job is to make sure each number is controlled, tested and reproducible. The writing and judgment about what a result actually means stay central, since agents and people both lean on these evaluations to choose tools. Careful statistics and clear graphs become the trust layer that AI driven decisions rest on.

How it builds AI experience

The role builds direct experience designing and running evaluations of AI tools and agents, including grading rules and judge setup. It also grows skill in spotting where evaluations fail and in communicating results to technical audiences. Working alongside researchers on real benchmark data is the practical path to that experience.

Worth showing
  • Mixed effects modeling
  • Clear technical writing
  • Reproducible charts in code
  • Spotting weak results

This is our own read, written from the employer's posting and the company's Spotlight. The employer's posting is the source of truth for the role.

Openbenchmarks runs independent, open-source benchmarks that compare APIs, tools and agents on the same tasks under the same conditions, per the company.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Benchmarks Storyteller
Benchmarks Storyteller

OpenFunnel (YC F24) • San Francisco (CA)

On-site
USD 120,000 - 180,000
Benchmark Insight Engineer
Benchmark Insight Engineer

present of AI • United States

On-site
USD 90,000 - 150,000
Founding GTM
Founding GTM

OpenFunnel (YC F24) • San Francisco (CA)

On-site
USD 90,000 - 130,000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

snorkelai • United States

Remote
USD 120,000 - 180,000
Research Lead, Open Benchmarks Grants
Research Lead, Open Benchmarks Grants

snorkelai • United States

Remote
USD 231,000 - 390,000
Analyst, Applied Validations
Analyst, Applied Validations

present of AI • New York (NY)

On-site
USD 120,000 - 180,000
Member Of Technical Staff — AI Systems & Benchmarking
Member Of Technical Staff — AI Systems & Benchmarking

Aionia Group • San Francisco (CA)

On-site
USD 130,000 - 220,000
Equity
On-site
Benchmark Storyteller: Stats, Graphs & Clear Reports
Benchmark Storyteller: Stats, Graphs & Clear Reports

OpenFunnel (YC F24) • San Francisco (CA)

On-site
USD 120,000 - 180,000
AI Benchmark Task Author · Turing Turing · Varies · remote · 10h ago Varies 10h ago
AI Benchmark Task Author · Turing Turing · Varies · remote · 10h ago Varies 10h ago

Benture • San Francisco (CA)

Remote
USD 55,000 - 110,000
Expert Talent Pool, Evals and Benchmarks
Expert Talent Pool, Evals and Benchmarks

Clera • United States

Remote
USD 90,000 - 150,000