Member of Technical Staff (Evals)

Handshake; Mountain View (CA); On-site; USD 190,000 - 260,000; Full time

Benefits offered by this job

Health Insurance

401k match

Sabbatical

Learning & development stipend

Parental leave

Flexible time off

Volunteer time off

Job summary

Handshake seeks a Member of Technical Staff, Evals to define how frontier AI systems are measured, understood, and improved. You will partner with AI researchers, domain experts, and customers to develop benchmarks, reward systems, verifiers, and data-quality techniques.

Early members influence our technical direction, culture, and open-source software. We value demonstrated research capability and a builder’s mindset over titles or degree.

Qualifications

• PhD in ML/AI, CS, data science, or related fields or equivalent industry research experience.
• Publications at top AI/ML venues (NeurIPS, ICML, ICLR, COLM).
• Strong Python skills and experience building scalable software.
• Builders who ship benchmarks, datasets, or software with an OSS portfolio.
• Deep knowledge of frontier AI benchmarks, eval techniques, and agent harnesses.
• Active conference participation and public open-source contributions.

Responsibilities

• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and RL environments.
• Develop reward models, programmatic verifiers, graders, and data-quality techniques.
• Research what makes evaluations representative, reliable, and resistant to shortcutting.
• Build systems for high-quality human data with task design and data attribution.
• Run fast iteration loops: prototype, evaluate, interpret results, diagnose failures, and improve benchmarks.
• Publicly contribute to the field through benchmarks, open-source tools, and research writing.

Skills

Python

Research

Publications

Ownership

Agent development

Education

PhD in ML/AI or related field

Tools

GitHub

Open-source software

Job description

• We are hiring a Member of Technical Staff, Evals to help define how frontier AI systems are measured, understood, and improved
• This is a broad, high-ownership role for researchers who build
• You will partner with AI researchers, domain experts, and customers to develop new benchmarks, reward and verifier systems, agent-evaluation methodologies, and data-quality techniques
• You will work on questions at the center of frontier AI progress: what should be measured, how to design evaluations that reflect real capability, how to create high-signal feedback, and how to build the environments and data systems that make those answers actionable
• Early members of the team will have unusual influence over our technical direction, operating culture, and the open-source software, benchmarks, and research products we build
• We care more about demonstrated research capability, technical judgment, and a builder’s mindset than a specific title, degree, or career path
• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments
• Develop reward models, programmatic verifiers, graders, and other feedback systems that make model behavior measurable and improvable
• Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking
• Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques
• Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system
• Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing

Benefits

• Health Insurance: Comprehensive medical , dental, and vision coverage
• Fertility and family forming: Through a partnership, Handshake offers comprehensive family benefits, including infertility treatments and a $15K stipend, to its employees.
• LGTBQ+ coverage: All medical plans cover transgender services.
• Mental health: Discounts on a variety of wellness apps and an Employee Assistance Program with access to five free counseling sessions.
• Tax savings: Tax-advantaged benefits to save money on healthcare, dependent care, and other expenses.
• Life & disability insurance: Essential benefits and automatic coverage with basic life insurance, plus short-term and long-term disability coverage.
• 401k match: Dollar-for-dollar match up to 1% of pay, with a max of $1,200 per year.
• Equity: Our equity program includes grants at hire, refresher grants after two years of tenure, and grants alongside promotions.
• Financial management: Handshake partners to provide free professional financial planning services.
• Learning & development: Employees are provided with an annual stipend to use towards professional development, career development workshops, manager trainings, and 1-on-1 coaching through external partnerships
• Community leader support: Employee Resource Group leaders are compensated with a quarterly bonus for the high-impact work the role requires
• Parental leave: Leaves of 16 weeks for birth-giving parents and 10 weeks for non-birth-giving parents; the policy applies to adoptions as well. Handshake also partners to provide one-on-one coaching on preparing for leave and coming back from leave
• Flexible time off: No accrued time off or waiting period to take vacation. Two paid flexible holiday days off to observe holidays that closely align with religious, cultural, and personal needs
• Sabbatical: Paid six-week sabbatical to recharge and reset beginning at six years of continuous, full-time employment, and every four years thereafter
• Volunteer time off: Two days off for social and civil activism, volunteer work, and/or any related activities, including attending and participating in protests

Requirements

• Comfort operating in an ambiguous, fast-moving environment with substantial ownershipPhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry)
• Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM
• Strong Python skills, experience building scalable software, working with agents
• Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets (e.g. a strong GitHub profile / OSS contributions, or product portfolio)
• Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes
• Join a rapidly growing company whose data business grew from zero to nearly $1B run rate in a year
• Help build an early technical organization where your work shapes the roadmap, standards, and culture
• Work at the very frontier of AI with most major AI labs, researching some of the most important problems in Data and Evaluations
• Attend (and publish at) conferences like NeurIPS, ICML, ICLR, COLM
• Publish results and work in public through open-source benchmarks and software, papers, and blogs