Senior Benchmark Engineer - Public Data Provider Evals

Firecrawl; Toronto; Hybrid; CAD 210,000 - 290,000; Full time

Benefits offered by this job

Salary that makes sense

Own a piece

Generous PTO

Parental leave

Wellness stipend

Learning & Development

Team offsites

Sabbatical

Job summary

Firecrawl is seeking a Research Engineer to design and run head-to-head benchmarks of data providers, establishing ground truth, and shipping a reproducible weekly release. You’ll evaluate accuracy, coverage, freshness, speed, and cost while collaborating with marketing to publish public leaderboards.

You’ll own the test datasets, scoring methods, and the loop into Alexandria. This role requires 4+ years in ML, data engineering, or benchmark work and a track record of shipping tangible evals in

Qualifications

• You’ve shipped evals or benchmarks, and you can explain exactly why your results were trustworthy.
• You’ve done this somewhere that matters: a frontier lab, a data company like Scale, Surge, micro1 or Mercor, a third-party benchmark org, or a public benchmark project. OSS contributors very welcome.
• You’re strong in Python and API integrations, and comfortable with messy vendor APIs, rate limits, and inconsistent schemas.
• You know how to build test sets and scoring methods: sampling, labeling, inter-rater agreement, and when to trust an LLM judge and when not to.
• You use AI heavily and keep upgrading your own workflow. You ship without waiting for instructions.
• You write findings clearly for engineers, marketers, and the vendors on the other side of the leaderboard.

Responsibilities

• Design and run head-to-head benchmarks of data providers against verified ground truth. For example:
• Build and maintain the test datasets and ground truth, and keep them from going stale or leaking.
• Own the automated harness and the weekly benchmark release. Every run has to be reproducible, versioned, and defensible.
• Measure what buyers actually care about: accuracy, coverage, freshness, speed, and cost.
• Work with marketing to ship public leaderboard pages that are useful and hold up under scrutiny.
• Close the loop into Alexandria so benchmark results change which provider gets called for what.
• Keep expanding the categories we benchmark, inside Alexandria and beyond it.

Skills

Python

API integrations

Benchmarking

Data engineering

Documentation

Tools

Git

Testing harness

Job description

Firecrawl is seeking a Research Engineer to design and run head-to-head benchmarks of data providers, establishing ground truth, and shipping a reproducible weekly release. You’ll evaluate accuracy, coverage, freshness, speed, and cost while collaborating with marketing to publish public leaderboards.

You’ll own the test datasets, scoring methods, and the loop into Alexandria. This role requires 4+ years in ML, data engineering, or benchmark work and a track record of shipping tangible evals in