Staff Engineer, Frontier AI Evaluations & Benchmarks

Handshake; New York (NY); Hybrid; USD 200,000 - 350,000; Full time

Benefits offered by this job

Equity

401(k) match

Parental benefits

Medical/dental/vision

Mental health support

Wellness stipend

Learning stipend

Commuting support

Free lunch

Gym access

Flexible PTO

Holidays & flex days

Team outings

Job summary

Handshake is redefining how frontier AI systems are evaluated, measured, and improved. This role collaborates with AI researchers, domain experts, and customers to define benchmarks, reward and verifier systems, and data-quality techniques.

Early team members influence technical direction and open-source outputs. The ideal candidate holds a PhD or equivalent research track record, with publications in top venues and a proven ability to ship high-quality software, benchmarks, or datasets.

Qualifications

• PhD in ML/AI or related field or equivalent research experience in industry.
• Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
• Builders who ship high-quality software, benchmarks, or datasets.
• Strong Python skills and ability to build scalable software.
• Experience with frontier AI benchmarks, eval techniques, and data systems.

Responsibilities

• Design and build evaluation frameworks and benchmarks for frontier LLMs, AI agents, multimodal models, and RL environments.
• Develop reward models, verifiers, graders, and other feedback systems.
• Research evaluation representativeness and reliability; prevent shortcutting and reward hacking.
• Build systems for high-quality human data, including task design and data-quality signals.
• Run fast iteration loops: prototype, evaluate, diagnose failures, and push next benchmarks.
• Contribute to the field through open-source tools, research, and technical writing.

Skills

Python

Research

OSS contributions

Agent development

Publications

Education

PhD in ML/AI or related

Tools

GitHub

Job description

Handshake is redefining how frontier AI systems are evaluated, measured, and improved. This role collaborates with AI researchers, domain experts, and customers to define benchmarks, reward and verifier systems, and data-quality techniques.

Early team members influence technical direction and open-source outputs. The ideal candidate holds a PhD or equivalent research track record, with publications in top venues and a proven ability to ship high-quality software, benchmarks, or datasets.