Staff Engineer, AI Benchmarking & Agent Evaluation

Handshake; Mountain View (CA); On-site; USD 270,000 - 350,000; Full time

Benefits offered by this job

Equity

401(k) match

Parental leave

Wellbeing stipend

Learning stipend

Office gym

Free lunch

Flexible PTO

Holidays

Referral bonuses

Job summary

Handshake is hiring a Member of Technical Staff to define how frontier coding agents are benchmarked and improved. You will collaborate with world-class AI researchers and software engineers to develop new benchmarks, reward systems, verifier methodologies, and data-quality techniques focused on AI for coding.

Early team members will influence technical direction, culture, and open-source software, benchmarks, and research products.

Qualifications

• Experience building or evaluating coding benchmarks.
• Published research on AI for code generation or related areas.
• Extensive software engineering experience with OSS contributions and knowledge of agent harnesses.

Responsibilities

• Create/publish Coding benchmarks that challenge state-of-the-art frontier agents.
• Understand/shape the frontier of coding agents and where the space is headed, publishing studies of impactful findings.
• Help the world’s top AI Labs improve their models for coding.
• Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.

Skills

Coding benchmarks

Agent harnesses

OSS contributions

Tools

GitHub

Job description

Handshake is hiring a Member of Technical Staff to define how frontier coding agents are benchmarked and improved. You will collaborate with world-class AI researchers and software engineers to develop new benchmarks, reward systems, verifier methodologies, and data-quality techniques focused on AI for coding.

Early team members will influence technical direction, culture, and open-source software, benchmarks, and research products.