Staff Scientist, Frontier AI Evaluations

Handshake; United States; Hybrid; USD 200,000 - 350,000; Full time

Benefits offered by this job

Ownership equity in a fast-growing公司

401(k) match & competitive comp.

Paid parental leave & fertility-benefi

Medical, dental, vision, mental health

$2,000 learning stipend

Commuting support, free lunch, SF gym

Flexible PTO + holidays

Team outings & referral bonuses

Job summary

Handshake is hiring a Member of Technical Staff, Evals to define how frontier AI systems are measured and improved. You will partner with AI researchers, domain experts, and customers to develop benchmarks, reward systems, and data-quality techniques.

Location: San Francisco preferred; exceptional candidates in other locations are welcome. Early members influence direction, culture, and OSS assets we build.

Qualifications

• PhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry).
• Publications at top AI/ML venues (e.g., NeurIPS, ICML, ICLR, COLM).
• Strong Python skills and experience building scalable software, including agents.
• Builders who ship high-quality software, benchmarks, or datasets with OSS contributions.

Responsibilities

• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and RL environments.
• Develop reward models, programmatic verifiers, graders, and other feedback systems to measure and improve model behavior.
• Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking.
• Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
• Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system.
• Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.

Skills

Python

Research

Software engineering

GitHub / OSS contributions

Benchmarks

Ambiguity tolerance

Education

PhD in ML/AI, CS, or data science

Tools

GitHub

Job description

Handshake is hiring a Member of Technical Staff, Evals to define how frontier AI systems are measured and improved. You will partner with AI researchers, domain experts, and customers to develop benchmarks, reward systems, and data-quality techniques.

Location: San Francisco preferred; exceptional candidates in other locations are welcome. Early members influence direction, culture, and OSS assets we build.