AI Evaluation Architect — Frontier Benchmarks & Systems

Handshake; Mountain View (CA); On-site; USD 200,000 - 350,000; Full time

Benefits offered by this job

Ownership: Equity

401(k) match

Parental leave

Wellbeing benefits

Learning stipend

Commuting support & free lunch

Gym in SF office

Flexible PTO

Holidays + flex days

Job summary

Handshake seeks a Member of Technical Staff, Evals to define how frontier AI systems are measured, understood, and improved. You will partner with AI researchers and customers to develop benchmarks, evaluators, and data-quality techniques, influencing our technical direction and open-source products.

Location: San Francisco preferred; exceptional candidates in other locations considered. You will design evaluation frameworks, build high-signal feedback loops, and publicly share results through

Qualifications

• PhD in ML/AI, CS, data science, or related field.
• Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
• Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets.
• Strong Python skills and experience building scalable software.
• Familiarity with frontier AI benchmarks, eval techniques, and data-quality concepts.

Responsibilities

• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments.
• Develop reward models, programmatic verifiers, graders, and other feedback systems that make model behavior measurable and improvable.
• Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking.
• Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
• Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system.
• Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.

Skills

Python

Publications

OSS contributions

Research-minded

Education

PhD in ML/AI or related field

Tools

GitHub

Open-source benchmarks

Job description

Handshake seeks a Member of Technical Staff, Evals to define how frontier AI systems are measured, understood, and improved. You will partner with AI researchers and customers to develop benchmarks, evaluators, and data-quality techniques, influencing our technical direction and open-source products.

Location: San Francisco preferred; exceptional candidates in other locations considered. You will design evaluation frameworks, build high-signal feedback loops, and publicly share results through