Research Engineer, AI Benchmarks for Frontier Agents

HUD; Singapore; Remote; SGD 100,000 - 175,000; Full time

Benefits offered by this job

Medical coverage

Lunch and dinner for in-office staff

Holiday break

401k match

Equinox membership

Commuter benefits

Unlimited access to tokens for AI APIs

Top-floor offices in SF and Singapore

Annual travel budget to visit offices

Job summary

HUD is seeking Research Engineers to build high-quality benchmarks for evaluating frontier agents on domain-specific tasks. You’ll design, implement, and maintain benchmarks that are technically rigorous, practical, and credible to frontier labs.

You will collaborate with experts to define tasks, build infrastructure to run models, and develop metrics to assess benchmark reliability, with an emphasis on real-world relevance and clear documentation for a technical audience.

Qualifications

• Proficiency in Python, Docker, and Linux environments.
• Published papers or technical blogs on benchmarks and model failure modes; link in application.
• Strong understanding of what a good benchmark means and what makes one realistic, reliable, and useful.
• Experience working on environments and evals.
• Curiosity to understand workflows across domains.

Responsibilities

• Own the design, implementation, and quality of HUD’s internal agent benchmarks
• Work with subject-matter experts to define tasks and create domain-specific benchmarks
• Build infrastructure to reliably run models and agents against benchmark tasks
• Develop metrics and analyses to understand benchmark difficulty, reliability, and failure modes
• Validate benchmark performance against real-world evals, customer needs, and lab expectations
• Write clear documentation and benchmark reports for technical audiences

Skills

Python

Linux

Tools

Docker

Job description

HUD is seeking Research Engineers to build high-quality benchmarks for evaluating frontier agents on domain-specific tasks. You’ll design, implement, and maintain benchmarks that are technically rigorous, practical, and credible to frontier labs.

You will collaborate with experts to define tasks, build infrastructure to run models, and develop metrics to assess benchmark reliability, with an emphasis on real-world relevance and clear documentation for a technical audience.