Eval Infrastructure Engineer — Define LLM Quality Metrics

Firecrawl; San Francisco (CA); Hybrid; USD 160,000 - 240,000; Full time

Benefits offered by this job

Unlimited PTO

12 weeks fully paid parental leave

Wellness stipend

Learning & Development budget

3 paid months sabbatical after 4 years

Medical, dental, and vision insurance

401(k) plan

Pet insurance

Job summary

Firecrawl is seeking a Research Engineer for Evals to build the evaluation systems that ensure the quality of outputs. This role involves designing metrics, owning feedback loops, and developing robust evaluation frameworks. The candidate should have a strong ML engineering background with at least 3 years of experience and a firm grasp on data quality. The role offers a salary range of $160,000 to $240,000 and allows for remote work across the Americas, contributing to a dynamic and fast-paced technological environment.

Qualifications

• 3+ years of experience in ML engineering, applied AI, or data quality.
• Proven experience building evaluation frameworks.
• Strong background in working with unstructured web data.

Responsibilities

• Build the evaluation stack from scratch to measure output quality.
• Design benchmarks reflecting real unstructured web data.
• Own automated evaluation systems for LLM output.

Skills

ML engineering

Data quality

Pipeline design

Metric definition

Labeled dataset curation

Job description

Firecrawl is seeking a Research Engineer for Evals to build the evaluation systems that ensure the quality of outputs. This role involves designing metrics, owning feedback loops, and developing robust evaluation frameworks. The candidate should have a strong ML engineering background with at least 3 years of experience and a firm grasp on data quality. The role offers a salary range of $160,000 to $240,000 and allows for remote work across the Americas, contributing to a dynamic and fast-paced technological environment.