Staff Scientist, Frontier AI Evaluations & Benchmarks

Openkrill; United States; Remote; USD 180,000 - 320,000; Full time

Benefits offered by this job

Equity in a fast-growing company

401(k) match

Paid parental leave

Fertility benefits

Mental health support

$2,000 learning stipend

Commuting support, SF office gym

Flexible PTO

Holidays + flex days

Team outings & referral bonuses

Job summary

Handshake is hiring a Member of Technical Staff, Evals to shape how frontier AI systems are measured, understood, and improved. You’ll collaborate with researchers, domain experts, and customers to craft new benchmarks, reward systems, and data-quality techniques.

Location: San Francisco preferred; exceptional candidates in other locations considered. The role offers high ownership and opportunities to influence technical direction and open-source projects.

Qualifications

• PhD in ML/AI, computer science, data science, or related fields (or equivalent industry research).
• Publications in top AI/ML venues (NeurIPS, ICML, ICLR, COLM).
• Builders who ship high-quality software, benchmarks, or datasets (strong OSS portfolio).
• Strong Python skills and experience building scalable software with agents.

Responsibilities

• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs and AI agents.
• Develop reward models, verifiers, graders, and other feedback systems.
• Research representative, difficult, reliable evaluations resistant to shortcutting.
• Build systems for high-quality human data, including task design and data-quality signals.
• Run fast iteration loops: prototype, evaluate, diagnose failures, and improve benchmarks.
• Publicly contribute to the field through benchmarks, OSS, research, and writing.

Skills

Python

Agent experimentation

Evaluation frameworks

Frontier AI knowledge

Publications

Education

PhD in ML/AI or related field

Tools

GitHub OSS

Benchmarking tools

Job description

Handshake is hiring a Member of Technical Staff, Evals to shape how frontier AI systems are measured, understood, and improved. You’ll collaborate with researchers, domain experts, and customers to craft new benchmarks, reward systems, and data-quality techniques.

Location: San Francisco preferred; exceptional candidates in other locations considered. The role offers high ownership and opportunities to influence technical direction and open-source projects.