Lead, AI Safety Benchmarks & Evaluations

Alice (Formerly ActiveFence); United States; On-site; USD 180,000 - 280,000; Full time

Job summary

Alice (Formerly ActiveFence) is seeking a senior leader to drive safety and evaluation research for frontier AI models. You will own the taxonomy, oversee the benchmark suite, and coordinate a small pool of freelancers while aligning with the CTO's public research agenda.

You will work with 150 researchers, travel to conferences, and shape quarterly release plans based on client asks and emerging events in AI safety and security.

Qualifications

• PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research.
• 3+ years building and running safety or security evaluations for language models in production or research settings.
• 5+ relevant research publications in AI safety and security including lead author on at least 2 of them.
• Strong engineering background: evaluation harnesses, distributed inference, vLLM, and reading/fixing codebases.
• You can build a taxonomy, not only score against one.

Responsibilities

• Ship the benchmark cadence and manage the schedule for frontiers evaluation cycles.
• Own the quality bar: ensure clear rubrics, sane distributions, and taxonomy alignment.
• Run the process: hold timelines, coordinate researchers, and direct freelancers when needed.
• Set the roadmap with the forum: collaborate with CTO, pod, and leads to refresh quarterly plans.
• Stay ahead of the curve: engage with labs, researchers, and conferences; weekly contact.

Job description

Alice (Formerly ActiveFence) is seeking a senior leader to drive safety and evaluation research for frontier AI models. You will own the taxonomy, oversee the benchmark suite, and coordinate a small pool of freelancers while aligning with the CTO's public research agenda.

You will work with 150 researchers, travel to conferences, and shape quarterly release plans based on client asks and emerging events in AI safety and security.