Research Lead, AI Safety Benchmarks & Evaluations

Alice; New York (NY); On-site; USD 180,000 - 250,000; Full time

Job summary

Alice is seeking a Research Lead to drive evaluations and benchmarks in a fast‑moving AI safety context. You will own the taxonomy, harness, and release cycles, coordinating with in-house researchers and a budget for freelancers.

The role sits in the CTO office with ~150 researchers focusing on harms and safety across models and platforms. You will ship biweekly benchmarks, ensure rigorous quality standards, and manage a small pool of SME freelancers.

Qualifications

• PhD or Masters in computer science, machine learning or related field, or equivalent depth from industry.
• 3+ years building and running safety or security evaluations for language models in production.
• 5+ relevant research publications in AI safety and security including lead author on at least 2 of them
• Strong engineer: evaluation harnesses, distributed inference, vLLM, reading a codebase and fixing it
• You can build a taxonomy, not only score against one.
• You can direct a researcher and two freelancers without managing them formally.
• Strong English, written and spoken across time zones.
• Curiosity about the harms themselves: learn a new subject every three weeks.

Responsibilities

• Ship a benchmark cadence every two to three weeks.
• Own the quality bar: verifiers, rubrics, and taxonomy are clear and robust.
• Direct two to three freelancers (SMEs) ad-hoc when needed.
• Set the roadmap with the CTO, pod and research leads; align with accounts.
• Stay ahead of the curve by reading research, maintaining lab contacts, and attending conferences.

Skills

Strong English

Leadership

Research coordination

Budget management

Freelancer management

Education

PhD or Masters in computer science, ML or related field

Tools

Evaluation harnesses

Distributed inference

vLLM

Job description

Alice is seeking a Research Lead to drive evaluations and benchmarks in a fast‑moving AI safety context. You will own the taxonomy, harness, and release cycles, coordinating with in-house researchers and a budget for freelancers.

The role sits in the CTO office with ~150 researchers focusing on harms and safety across models and platforms. You will ship biweekly benchmarks, ensure rigorous quality standards, and manage a small pool of SME freelancers.