Research Lead, Open Benchmarks Grants

snorkelai; United States; Remote; USD 231,000 - 390,000; Full time

Job summary

Snorkel AI seeks a leader to drive research partnerships and funding decisions focused on open benchmarks for evaluating advanced AI systems. You will identify gaps in evaluation methods, assess proposals, and help collaborators develop rigorous benchmarks that measure meaningful capabilities or risks.

The work blends research judgment with hands-on technical investigation and collaboration with researchers at independent teams and AI labs.

Qualifications

• Requirements include a strong research record in evaluation, with leadership or substantial contribution to benchmarks, evaluation methods, or research environments used by researchers or model-development teams.
• Sound technical judgment about measurement quality, experimental rigor, and alternative explanations for results.
• Ability to design experiments to resolve uncertainty and recognize limitations that should change a research plan.
• Hands-on technical ability to build experiments, inspect tasks and evaluation harnesses, debug verifiers, and analyze model behavior.
• Experience guiding research collaborators through difficult technical decisions and helping bring projects to completion.
• Clear technical writing, shown through papers, artifacts, specifications, or reviews.

Responsibilities

• Identify important gaps in how advanced AI systems are evaluated and help set research priorities.
• Review grant proposals with research colleagues and work with prospective collaborators to strengthen their plans.
• Help determine which proposed projects should receive funding through an open benchmark grants program.
• Partner with funded researchers and fellows to define what a benchmark measures and design its tasks and experiments.
• Examine benchmark implementations, task harnesses, verifiers, and model behavior; investigate whether results support the claims being made.
• Work with AI lab researchers to understand their measurement needs, support benchmark adoption and interpretation, and incorporate findings into future research.
• Turn research outcomes into clear artifacts that other researchers can use and build upon.

Job description

Role overview

Lead research partnerships and funding decisions focused on open benchmarks for evaluating advanced AI systems. You will identify gaps in current evaluation methods, assess proposals, and help collaborators develop rigorous benchmarks that measure meaningful capabilities or risks. The work combines research judgment, hands-on technical investigation, and collaboration with researchers at independent teams and AI labs.

Responsibilities

• Identify important gaps in how advanced AI systems are evaluated and help set research priorities.
• Review grant proposals with research colleagues and work with prospective collaborators to strengthen their plans.
• Help determine which proposed projects should receive funding through an open benchmark grants program.
• Partner with funded researchers and fellows to define what a benchmark measures and design its tasks and experiments.
• Examine benchmark implementations, task harnesses, verifiers, and model behavior; investigate whether results support the claims being made.
• Work with AI lab researchers to understand their measurement needs, support benchmark adoption and interpretation, and incorporate findings into future research.
• Turn research outcomes into clear artifacts that other researchers can use and build upon.

Requirements

• A strong research record in evaluation, including leading or substantially contributing to benchmarks, evaluation methods, or research environments used by researchers or model-development teams.
• Sound technical judgment about measurement quality, experimental rigor, and alternative explanations for results.
• Ability to design experiments that resolve uncertainty and recognize limitations that should change a research plan.
• Hands-on technical ability to build experiments, inspect tasks and evaluation harnesses, debug verifiers, and analyze model behavior.
• Experience guiding research collaborators through difficult technical decisions and helping bring projects to completion.
• Clear technical writing, shown through papers, research artifacts, specifications, or reviews.

Benefits and work setup

The source describes opportunities to shape research priorities, contribute to strategic decisions, deepen technical expertise, and develop skills across functions. Compensation depends on qualifications, experience, skills, and location; the listed range is $231,000-$390,000.