AI Safety Benchmark Lead & Evaluation Architect

Alice; New York (NY); On-site; USD 180,000 - 250,000; Full time

Job summary

Alice is seeking a seasoned researcher-leader to helm AI safety benchmarks and evaluations. You will own the taxonomy, harness, and release cadence, coordinating researchers and freelancers while reporting to the CTO office.

You will influence the public research agenda and shape quarterly plans with cross-functional leads. You will collaborate with top AI labs and universities, ensure rigorous evaluation standards, and mentor a growing team of researchers.

Qualifications

• Strong English, written and spoken. This role runs on internal communication across time zones.
• You can build a taxonomy, not only score against one.
• You can direct a researcher and two freelancers without managing them formally.
• Post-training experience: SFT, DPO, GRPO (ideally).
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on a client call; strong communication. Travel to conferences at least 3 times a year.

Responsibilities

• Ship the benchmark cadence every two to three weeks; adjust size by subject.
• Own the quality bar; ensure rubrics, verifiers, and taxonomy alignment.
• Run the process; keep researchers on timeline and manage ad-hoc freelancers.
• Set the roadmap with the forum; quarterly release plans aligned to accounts.
• Stay ahead of the curve by engaging labs, reading research, and attending conferences.

Skills

English proficiency

Research leadership

Experiment design

Taxonomy building

People coordination

Communication (verbal)

Public speaking

Education

PhD or MSc in CS/ML or related field

Job description

Alice is seeking a seasoned researcher-leader to helm AI safety benchmarks and evaluations. You will own the taxonomy, harness, and release cadence, coordinating researchers and freelancers while reporting to the CTO office.

You will influence the public research agenda and shape quarterly plans with cross-functional leads. You will collaborate with top AI labs and universities, ensure rigorous evaluation standards, and mentor a growing team of researchers.