Staff AI Evals Engineer for Frontier Benchmarks

Cacheflow; United States; On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity

401(k) match

Parental leave

Fertility benefits

Mental health support

Wellness stipend

Learning stipend

Free lunch

SF office gym

Commuting support

Flexible PTO

Holidays + flex days

Team outings

Referral bonuses

Job summary

Handshake is hiring a Member of Technical Staff, Evals to shape how frontier AI systems are measured and improved. You will work with AI researchers, domain experts, and customers to build benchmarks, reward/verifier systems, and data-quality techniques.

Early members will influence technical direction, open-source software, and research products. Location: San Francisco preferred; exceptional candidates in other locations are welcome.

Qualifications

• PhD in ML/AI, computer science, data science, or related field (or equivalent research experience).
• Publications at top AI/ML venues (NeurIPS, ICML, ICLR, COLM).
• Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets.
• Strong Python skills and experience building scalable software with agents.
• Knowledge of frontier AI: benchmarks, evaluation techniques, agent harnesses, data shapes.
• Comfort operating in an ambiguous, fast-moving environment with substantial ownership.

Responsibilities

• Design and build evaluation frameworks and benchmarks for frontier LLMs and agents.
• Develop reward models, programmatic verifiers, graders, and feedback systems.
• Research what makes evaluations reliable and resistant to shortcutting or reward hacking.
• Build systems for high-quality human data, including task design and data-quality signals.
• Run fast iteration loops: prototype, evaluate, interpret results, diagnose failures.
• Publicly contribute to the field through benchmarks, open-source tools, and research papers.

Skills

Python

Research

Software development

AI/ML knowledge

Education

PhD in ML/AI or related field

Tools

GitHub / OSS contributions

Python tooling

Job description

Handshake is hiring a Member of Technical Staff, Evals to shape how frontier AI systems are measured and improved. You will work with AI researchers, domain experts, and customers to build benchmarks, reward/verifier systems, and data-quality techniques.

Early members will influence technical direction, open-source software, and research products. Location: San Francisco preferred; exceptional candidates in other locations are welcome.