Evaluation Systems Engineer: Metrics, Datasets & Pipelines

Visa Hunt; San Francisco (CA); Hybrid; USD 210,000 - 275,000; Full time

Benefits offered by this job

Salary and equity

Generous PTO

Parental leave

Wellness stipend

Learning & Development

Team offsites

Sabbatical

SF HQ perks

Job summary

Firecrawl is seeking a Research Engineer to design evaluation systems that determine if our URL-to-data pipeline works across millions of sites. You’ll define metrics, build the measurement pipelines, generate datasets, and own the feedback loop from output quality back to model and product decisions.

Requires 4+ years in ML, research engineering, or data-heavy backend with hands-on evaluation work. Located in San Francisco with hybrid/on-site requirements; visa is not sponsored at this time.

Qualifications

• 4+ years in ML, research engineering, or data-heavy backend with real evaluation work
• Ability to design metrics and measurement systems
• Experience building scalable evaluation pipelines and datasets

Responsibilities

• Design the metrics that define what 'good output' means across sites and formats
• Build the pipelines and harnesses that measure quality at scale
• Generate and curate the datasets that support trustworthy evaluation
• Own the feedback loop from output quality back to model and product decisions
• Turn 'did that work?' into actionable insights for the team

Skills

Metrics design

Pipeline building

Dataset generation

Quality feedback loop

Attention to edge cases

Job description

Firecrawl is seeking a Research Engineer to design evaluation systems that determine if our URL-to-data pipeline works across millions of sites. You’ll define metrics, build the measurement pipelines, generate datasets, and own the feedback loop from output quality back to model and product decisions.

Requires 4+ years in ML, research engineering, or data-heavy backend with hands-on evaluation work. Located in San Francisco with hybrid/on-site requirements; visa is not sponsored at this time.