Lead Open Benchmark Grants & Evaluation Research

snorkelai; Stati Uniti; Remoto; USD 231.000 - 390.000; Tempo pieno

Descrizione del lavoro

Snorkel AI seeks a leader to drive research partnerships and funding decisions focused on open benchmarks for evaluating advanced AI systems. You will identify gaps in evaluation methods, assess proposals, and help collaborators develop rigorous benchmarks that measure meaningful capabilities or risks.

The work blends research judgment with hands-on technical investigation and collaboration with researchers at independent teams and AI labs.

Competenze

• Requirements include a strong research record in evaluation, with leadership or substantial contribution to benchmarks, evaluation methods, or research environments used by researchers or model-development teams.
• Sound technical judgment about measurement quality, experimental rigor, and alternative explanations for results.
• Ability to design experiments to resolve uncertainty and recognize limitations that should change a research plan.
• Hands-on technical ability to build experiments, inspect tasks and evaluation harnesses, debug verifiers, and analyze model behavior.
• Experience guiding research collaborators through difficult technical decisions and helping bring projects to completion.
• Clear technical writing, shown through papers, artifacts, specifications, or reviews.

Mansioni

• Identify important gaps in how advanced AI systems are evaluated and help set research priorities.
• Review grant proposals with research colleagues and work with prospective collaborators to strengthen their plans.
• Help determine which proposed projects should receive funding through an open benchmark grants program.
• Partner with funded researchers and fellows to define what a benchmark measures and design its tasks and experiments.
• Examine benchmark implementations, task harnesses, verifiers, and model behavior; investigate whether results support the claims being made.
• Work with AI lab researchers to understand their measurement needs, support benchmark adoption and interpretation, and incorporate findings into future research.
• Turn research outcomes into clear artifacts that other researchers can use and build upon.

Descrizione del lavoro

Snorkel AI seeks a leader to drive research partnerships and funding decisions focused on open benchmarks for evaluating advanced AI systems. You will identify gaps in evaluation methods, assess proposals, and help collaborators develop rigorous benchmarks that measure meaningful capabilities or risks.

The work blends research judgment with hands-on technical investigation and collaboration with researchers at independent teams and AI labs.