AI Evaluation Benchmark Fellow

Snorkel AI; New York, San Francisco (NY, CA); Hybrid; USD 117,000 - 227,000; Full time

Benefits offered by this job

Research advisor

Compute resources

Open-source artifact

Community events

Job summary

Snorkel AI is seeking a Research Fellow to advance open benchmarks and evaluation environments that shape how AI progress is measured. You will own the project from research question to public release, with a Snorkel co-advisor and access to compute and domain experts.

You’ll spend time between your academic affiliation and Snorkel’s research team, with opportunities to work in our San Francisco or New York offices where applicable.

Qualifications

• PhD candidates or postdocs with a strong research background in AI, evaluation, or related fields
• Ability to turn research ideas into working code, datasets, or tools
• Experience collaborating with domain experts to turn judgments into specifications
• Capacity to scope a project to a public release within the fellowship term

Responsibilities

• Scope and lead a new benchmark, dataset, or environment in a domain where current evaluations fall short
• Define the task specification, rubrics, and reference answers with domain experts from Snorkel's network, and run the expert review process
• Release the research artifact publicly, with you as lead author
• Join weekly research meetings to share progress, receive feedback, and contribute to other researchers’ projects

Job description

Snorkel AI is seeking a Research Fellow to advance open benchmarks and evaluation environments that shape how AI progress is measured. You will own the project from research question to public release, with a Snorkel co-advisor and access to compute and domain experts.

You’ll spend time between your academic affiliation and Snorkel’s research team, with opportunities to work in our San Francisco or New York offices where applicable.