Benchmark Engineer — Public Data Provider Leaderboard

Showcify, Inc.; United States; Hybrid; USD 250,000 - 290,000; Full time

Benefits offered by this job

Salary that makes sense: $250,000–$290

Generous equity

Generous PTO: 15 days +

parental leave: 12 weeks

Wellness stipend: $100 USD/month

Learning & Development: $1,000 USD/yr

Team offsites

Sabbatical: 3 paid months

Job summary

Firecrawl is seeking a Research Engineer to design and run independent benchmarks comparing data providers. You will own the datasets, ground truth, scoring, and weekly release, publishing results publicly.

The role requires strong Python and API integration skills, with a track record of shipping evaluation work in a frontier tech environment. You will measure accuracy, coverage, freshness, speed, and cost, and work with marketing to present leaderboards.

Qualifications

• Shipped evals or benchmarks with defendable results.
• Experience building test sets and scoring methods.
• Strong Python and API integrations skills.
• Ability to communicate findings clearly to engineers and vendors.
• Experience using AI to improve workflows.

Responsibilities

• Design and run head-to-head benchmarks of data providers against ground truth.
• Build and maintain test datasets and ensure they stay fresh.
• Own the automated harness and weekly benchmark release; ensure reproducibility.
• Measure accuracy, coverage, freshness, speed, and cost in benchmarks.
• Collaborate with marketing to publish public leaderboards.
• Integrate benchmark results into provider selection processes within Alexandria.
• Expand benchmark categories over time.

Skills

Python

APIs

Benchmarking

Job description

Firecrawl is seeking a Research Engineer to design and run independent benchmarks comparing data providers. You will own the datasets, ground truth, scoring, and weekly release, publishing results publicly.

The role requires strong Python and API integration skills, with a track record of shipping evaluation work in a frontier tech environment. You will measure accuracy, coverage, freshness, speed, and cost, and work with marketing to present leaderboards.