Research Fellow: Open AI Benchmarks & Evaluation

Snorkel AI; New York (NY); On-site; USD 117,000 - 227,000; Part time

Benefits offered by this job

Co-advisor

Compute access

Open-source artifact

Research events

Job summary

Snorkel AI is seeking a Research Fellow to build open benchmarks and evaluation environments that shape how AI progress is measured. You will own the project from the initial research question through public release, with a Snorkel researcher as co-advisor and access to compute and domain experts.

You’ll spend at least one day a week with Snorkel’s research team, collaborate in our San Francisco or New York offices where applicable, and publish an open-source artifact as part of your work.

Qualifications

• You’re excited to advance how AI is evaluated in your field and have the research background to design and build open benchmarks, datasets, or environments.
• You have demonstrated the ability to turn research ideas into working code, datasets, tools, or other usable research artifacts
• You work well with domain experts outside your own field and can turn their judgment into a specification
• You can scope a project to a public release within the fellowship term

Responsibilities

• Scope and lead a new benchmark, dataset, or environment in a domain where current evaluations fall short
• Define the task specification, rubrics, and reference answers with domain experts from Snorkel's network, and run the expert review process
• Release the research artifact publicly, with you as lead author.
• Join weekly research meetings to share progress, receive feedback, and contribute to other researchers’ projects

Job description

Snorkel AI is seeking a Research Fellow to build open benchmarks and evaluation environments that shape how AI progress is measured. You will own the project from the initial research question through public release, with a Snorkel researcher as co-advisor and access to compute and domain experts.

You’ll spend at least one day a week with Snorkel’s research team, collaborate in our San Francisco or New York offices where applicable, and publish an open-source artifact as part of your work.