Benchmarks Storyteller

OpenFunnel (YC F24)

San Francisco (CA)

Sur place

USD 120 000 - 180 000

Plein temps

Il y a 8 jours

Obtenez une réponse de cet employeur

Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.

Passez les filtres ATS

Résumé du poste

Openbenchmarks in San Francisco is seeking a data-savvy technical writer to document benchmark methodologies, results, and limitations, ensuring reproducibility. You will craft reports that describe the experiments with precision and clarity, enabling others to reproduce and trust the conclusions.

We expect a strong background in statistics, experience turning raw results into compelling visuals, and the ability to publish reports that withstand scrutiny.

Qualifications

  • Strong background in statistics: regression, mixed-effects models, significance testing, and understanding when results are not significant.
  • Experience designing or running evals (ML or LLM evaluation) and knowing where they go wrong.
  • Clear technical prose: methodology, results and limitations that a skeptical reader trusts.

Responsabilités

  • Write benchmark methodologies, results, limitations with enough detail to reproduce.
  • Present standard errors, confidence intervals, and explanations of outliers.
  • Design controlled comparisons and document grading rules and experimental setup so results are reproducible.
  • Create graphs and reports that communicate findings clearly to researchers and engineers.

Connaissances

Statistics
Experimental design
Technical writing
Data visualization

Description du poste

About Openbenchmarks

Agents are becoming first-class users of the internet. They research, compare tools and increasingly make build-versus-buy decisions on behalf of people. Every company will need its product picked by agents, and agents prefer open, independent, grounded benchmarks to make that call.

Agents are becoming first-class users of the internet. They research, compare tools and increasingly make build-versus-buy decisions on behalf of people. Every company will need its product picked by agents, and agents prefer open, independent, grounded benchmarks to make that call. Openbenchmarks is the evaluation infrastructure for agents: domain-specific, reproducible evaluations that help agents and people pick tools with confidence. Our mission is to be the trusted evaluation layer for agents. Founders previously led AI research and infra teams at Oracle and Appfolio. We're a small team of researchers and engineers working with fast-growing AI-first companies like Parallel, Firecrawl, Telnyx, TinyFish and more.

About The Role

Every benchmark we run produces a lot of data: accuracy, latency, cost, failure modes, and how results change between versions. You make sure what we publish is correct, clear and convincing, and you communicate it. That means rigorous statistics, graphs that show the real trade-off at a glance, and technical writing that holds up, from reports to posts. The bar is a benchmark report a skeptical researcher can't poke holes in. Every number has an error bar, every comparison is controlled, and every claim is backed by a test.

What You'll Work On

Technical writing - Write up each benchmark: methodology, results, limitations, and enough detail for anyone to reproduce our work. Technical reports by default, research papers when the work warrants it.

Statistics - Standard errors and confidence intervals on every result, stated sample sizes, and proper tests (e.g. mixed-effects models with per-task effects) before we call one tool better than another. Explain the outliers, don't hide them.

Methodology - Design controlled comparisons: isolate one variable, hold the rest fixed, and write up the grading rules and judge setup so anyone can reproduce the result.

Graphs - Score vs. cost with Pareto frontiers, cost broken into components, latency and cost distributions, and per-category results with error bars. Built in code, clear before pretty.

Communicating results - Turn each benchmark into reports, launch posts and threads for X, LinkedIn, Reddit and HN. Lead with one sharp finding and one graph.

Finding the story - Work with our researchers on the raw results. Dig into individual failures, spot what changed between runs, and decide what's worth saying.

You might be a fit if
  • You have a strong background in statistics: regression, mixed-effects models, significance testing, and you know when a result isn't significant
  • You've designed or run evals (ML or LLM evaluation) and know where they go wrong
  • You write clear technical prose: methodology, results and limitations that a skeptical reader trusts
  • You've made graphs that changed how people saw a result, and can show them
Nice to have
  • You've written research papers (published or not)
  • Familiarity with existing benchmarks (SWE-bench, Terminal-bench, BrowseComp) and how vendors cite them
  • An audience on X, LinkedIn or a blog
Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

Founding GTM
Founding GTM

OpenFunnel (YC F24) • San Francisco (CA)

Sur place
USD 90 000 - 130 000
Benchmarks Storyteller
Benchmarks Storyteller

present of AI • États-Unis

Sur place
USD 90 000 - 150 000
Benchmark Insight Engineer
Benchmark Insight Engineer

present of AI • États-Unis

Sur place
USD 90 000 - 150 000
Senior Researcher
Senior Researcher

Benchline Reports • États-Unis

Hybride
USD 90 000 - 150 000
Research Scientist, Open Benchmarks Red Team
Research Scientist, Open Benchmarks Red Team

snorkelai • États-Unis

À distance
USD 120 000 - 180 000
Senior Researcher
Senior Researcher

Staten Island Chamber of Commerce • New York (NY)

Hybride
USD 95 000 - 135 000
Remote, hybrid or on-site options
Office at 30 Hudson Yards (New York)
Small, growing team with mentorship
Benchmark Storyteller: Stats, Graphs & Clear Reports
Benchmark Storyteller: Stats, Graphs & Clear Reports

OpenFunnel (YC F24) • San Francisco (CA)

Sur place
USD 120 000 - 180 000
Founding GTM & Benchmark Content Lead
Founding GTM & Benchmark Content Lead

OpenFunnel (YC F24) • San Francisco (CA)

Sur place
USD 90 000 - 130 000
Research Engineer, Benchmarks
Research Engineer, Benchmarks

HUD • San Francisco (CA)

Sur place
USD 140 000 - 190 000
Competitive compensation
Top-tier medical, dental, and vision (
Lunch and dinner in office
+5
Software Engineer, Cloud Infrastructure
Software Engineer, Cloud Infrastructure

Benchling • États-Unis

Sur place
USD 140 000 - 200 000