Benchmark Research Engineer for Frontier AI

HUD; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Competitive compensation

Top-tier medical, dental, and vision (

Lunch and dinner in office

Holiday break & PTO

Equinox membership

401k

Commuter benefits

Unlimited tokens for AI tools

Job summary

HUD is seeking Research Engineers to build high-quality benchmarks for evaluating frontier agents on domain-specific tasks. You will own the design of HUD's internal agent benchmarks and work with experts to define tasks that mirror realistic workflows.

You will develop infrastructure to run models and agents against benchmarks, create metrics to assess difficulty and failure modes, and ensure results align with real-world needs and lab expectations, with clear documentation and reports.

Qualifications

• Proficiency in Python, Docker, and Linux environments.
• Published papers or technical blogs on benchmarks and their limitations, model failure modes, etc.
• Strong understanding of what a good benchmark means and what makes one realistic, reliable, and useful.
• Experience working on environments and evals.
• Curiosity and ability to understand workflows across domains.

Responsibilities

• Own the design, implementation, and quality of HUD’s internal agent benchmarks.
• Work with subject-matter experts to define tasks and create domain-specific benchmarks that evaluate agents on realistic workflows.
• Build infrastructure to reliably run models and agents against benchmark tasks.
• Develop metrics and analyses to understand benchmark difficulty, reliability, and failure modes.
• Validate whether benchmark performance correlates with real-world evals, customer needs, and lab expectations.
• Write clear documentation and benchmark reports that make results legible and credible to technical audiences.

Skills

Python

Research

Benchmark design

Eval frameworks

Cross-domain workflows

Education

Bachelor's degree in a relevant field

Tools

Docker

Linux

Job description

HUD is seeking Research Engineers to build high-quality benchmarks for evaluating frontier agents on domain-specific tasks. You will own the design of HUD's internal agent benchmarks and work with experts to define tasks that mirror realistic workflows.

You will develop infrastructure to run models and agents against benchmarks, create metrics to assess difficulty and failure modes, and ensure results align with real-world needs and lab expectations, with clear documentation and reports.