Researcher, Evaluations and Benchmarks

Alice; New York (NY); On-site; USD 180,000 - 250,000; Full time

Job summary

Alice is seeking a seasoned researcher-leader to helm AI safety benchmarks and evaluations. You will own the taxonomy, harness, and release cadence, coordinating researchers and freelancers while reporting to the CTO office.

You will influence the public research agenda and shape quarterly plans with cross-functional leads. You will collaborate with top AI labs and universities, ensure rigorous evaluation standards, and mentor a growing team of researchers.

Qualifications

• Strong English, written and spoken. This role runs on internal communication across time zones.
• You can build a taxonomy, not only score against one.
• You can direct a researcher and two freelancers without managing them formally.
• Post-training experience: SFT, DPO, GRPO (ideally).
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on a client call; strong communication. Travel to conferences at least 3 times a year.

Responsibilities

• Ship the benchmark cadence every two to three weeks; adjust size by subject.
• Own the quality bar; ensure rubrics, verifiers, and taxonomy alignment.
• Run the process; keep researchers on timeline and manage ad-hoc freelancers.
• Set the roadmap with the forum; quarterly release plans aligned to accounts.
• Stay ahead of the curve by engaging labs, reading research, and attending conferences.

Skills

English proficiency

Research leadership

Experiment design

Taxonomy building

People coordination

Communication (verbal)

Public speaking

Education

PhD or MSc in CS/ML or related field

Job description

You ship a benchmark every two to three weeks ( example benchmark ). Each one measures a frontier risk that nobody has measured yet. Some go public. Some go only to the labs.

Some of the benchmarks and papers are done in collaboration with the leading AI Labs and universities.

You will not write every eval yourself. Each benchmark pairs you with an in-house researcher who owns that harm area, and you get a budget for freelancers you direct. You own the taxonomy, the harness, the quality bar and the release.

The seat sits in the CTO office alongside the research lead who sets our public research agenda. Around 150 researchers here work on harms directly, and you can pull any of them onto a subject.

Key Responsibilities

1. Ship the benchmark cadence

A benchmark every two to three weeks. Size follows the subject. A chat-based taxonomy can carry 100 evals. An agentic or GRPO benchmark is closer to 20, because each one is expensive to read. Sensitivity decides what ships publicly and what goes to the labs alone.

2. Own the quality bar

The test is simple. A frontier lab reruns our set and gets our numbers. The verifiers hold, the rubrics are clear, the distribution is sane, and their subject matter experts read the taxonomy and call it novel.

That means you read the evals yourself. You can screen with a model, and you still open the file, spot the item that does not match the taxonomy, and push the researcher back on it.

3. Run the process

Hold the plan and the calendar. You are the person who keeps other researchers on timeline. You can direct two or three freelancers (SMEs) yourself ad-hoc when needed.

4. Set the roadmap with the forum

Roughly monthly you sit with the CTO and the pod and research leads. Inputs are what our research teams see, what clients are asking for, and what is moving in the news. Output is a revised release plan for the quarter, tied to the accounts we want to open.

5. Stay ahead of the curve

Around 20% of your time goes to the ecosystem. Read the research, keep contacts inside the labs, ask them what is bothering them, and travel to a couple of conferences a year. You should be talking to folks from the labs weekly.

Requirements

Requirements

• PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research.
• 3+ years building and running safety or security evaluations for language models in production, at an AI lab, a model provider, or a safety and security research organisation.
• 5+ relevant research publications in the field of AI safety and security including lead author on at least 2 of them
• Strong engineer. Evaluation harnesses, distributed inference, vLLM, reading a codebase and fixing it.
• You can build a taxonomy, not only score against one.
• You can direct a researcher and two freelancers without managing them formally.
• Strong English, written and spoken. This role runs on internal communication across time zones.
• Curiosity about the harms themselves. You will be learning a new subject every three weeks.

Ideally:

• Post-training experience: SFT, DPO, GRPO. Reward design for subjective and safety-relevant targets is a live problem for us.
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on a client call. Strong communication - both verbal and written, ability to present to large and/or senior audiences
• Travel to conferences at least 3 times a year

About Alice

Alice is a trust, safety, and security company built for the AI era. We safeguard the communicative technologies people use to create, collaborate, and interact- whether with each other or with machines.

Alice clients are the top 8 AI Labs in the world.

In a world where AI has fundamentally changed the nature of risk, Alice provides end-to-end coverage across the entire AI lifecycle. We support frontier model labs, enterprises, and UGC platforms with a comprehensive suite of solutions: from model hardening evaluations and pre-deployment red-teaming to runtime guardrails and ongoing drift detection.

Alice is widely considered a global leader in online safety and AI security. We have some of the most forward-thinking and passionate minds in the world working to safeguard over 3 billion users across the largest AI and tech platforms.

If you're creative and driven to secure the future of AI, we want to hear from you!

About the Position

You ship a benchmark every two to three weeks ( example benchmark ). Each one measures a frontier risk that nobody has measured yet. Some go public. Some go only to the labs.

Some of the benchmarks and papers are done in collaboration with the leading AI Labs and universities.

You will not write every eval yourself. Each benchmark pairs you with an in-house researcher who owns that harm area, and you get a budget for freelancers you direct. You own the taxonomy, the harness, the quality bar and the release.

The seat sits in the CTO office alongside the research lead who sets our public research agenda. Around 150 researchers here work on harms directly, and you can pull any of them onto a subject.

Key Responsibilities

1. Ship the benchmark cadence

A benchmark every two to three weeks. Size follows the subject. A chat-based taxonomy can carry 100 evals. An agentic or GRPO benchmark is closer to 20, because each one is expensive to read. Sensitivity decides what ships publicly and what goes to the labs alone.

2. Own the quality bar

The test is simple. A frontier lab reruns our set and gets our numbers. The verifiers hold, the rubrics are clear, and the distribution is sane, and their subject matter experts read the taxonomy and call it novel.

That means you read the evals yourself. You can screen with a model, and you still open the file, spot the item that does not match the taxonomy, and push the researcher back on it.

3. Run the process

Hold the plan and the calendar. You are the person who keeps other researchers on timeline. You can direct two or three freelancers (SMEs) yourself ad-hoc when needed.

4. Set the roadmap with the forum

Roughly monthly you sit with the CTO and the pod and research leads. Inputs are what our research teams see, what clients are asking for, and what is moving in the news. Output is a revised release plan for the quarter, tied to the accounts we want to open.

5. Stay ahead of the curve

Around 20% of your time goes to the ecosystem. Read the research, keep contacts inside the labs, ask them what is bothering them, and travel to a couple of conferences a year. You should be talking to folks from the labs weekly.

Requirements

Requirements

• PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research.
• 3+ years building and running safety or security evaluations for language models in production, at an AI lab, a model provider, or a safety and security research organisation.
• 5+ relevant research publications in the field of AI safety and security including lead author on at least 2 of them
• Strong engineer. Evaluation harnesses, distributed inference, vLLM, reading a codebase and fixing it.
• You can build a taxonomy, not only score against one.
• You can direct a researcher and two freelancers without managing them formally.
• Strong English, written and spoken. This role runs on internal communication across time zones.
• Curiosity about the harms themselves. You will be learning a new subject every three weeks.

Ideally:

• Post-training experience: SFT, DPO, GRPO. Reward design for subjective and safety-relevant targets is a live problem for us.
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on a client call. Strong communication - both verbal and written, ability to present to large and/or senior audiences
• Travel to conferences at least 3 times a year

About Alice

THE CHALLENGES ALONG THE WAY

1. Being Both Strategist and Executioner

One of the hardest parts of this role is that you’re both the visionary and the builder; the one drawing the mapandpaving the road.
That means switching between high-level strategy and hands-on experimentation daily, and doing it while bringing others along with you. There’s no playbook for this kind of work. You’re paving an unpaved road, one small experiment at a time.

2. Balancing Security and Innovation

ActiveFence is the leading provider of security and safety solutions for online experiences, safeguarding more than 3 billion users, top foundation models, and the world’s largest enterprises and tech platforms every day.
As a trusted ally to major technology firms and Fortune 500 brands that build user-generated and GenAI products, ActiveFence empowers security, AI, and policy teams with low-latency Real-Time Guardrails and a continuous Red Teaming program that pressure-tests systems with adversarial prompts and emerging threat techniques. Powered by deep threat intelligence, unmatched harmful-content detection, and coverage of 117+ languages, ActiveFence enables organizations to deliver engaging and trustworthy experiences at global scale while operating safely and responsibly across all threat landscapes.