Head of Evals

Trajectory Labs; Berkeley (CA); On-site; USD 250,000 - 400,000; Full time

Benefits offered by this job

Health coverage

401(k)

Catered meals

Relocation assistance

Job summary

Trajectory Labs in Berkeley, CA is building safety-focused evaluations for frontier AI models. You will own evaluations, review eval data, design new ones, and develop agent tooling to scale judgment.

Your work will sit at the intersection of safety research and engineering, with early-stage startup pace and autonomy to shape product and standards. Remote work is considered for the right candidate.

Qualifications

• Calibrated judgment to identify when a task, transcript, or environment is wrong and explain the evidence.
• Ability to build automation using LLMs and coding agents.
• Attention to detail across hundreds of reviews and tasks.
• Comfort with fast-paced, changing priorities in a startup.

Responsibilities

• Create evals to ensure data meets customer needs.
• Turn data into evals of customers’ models.
• Define evals strategy to scale data collection and testing.

Skills

Calibrated eval judgment

Agent-native engineering

Detail-oriented

Startup drive

Job description

Overview

You'll own evaluations across Trajectory. You’ll make sure that our AI safety data are ready to ship to our frontier lab customers, and you’ll build the evals that tell our customers how safe their models are.

For now, you'll mostly be directing agents rather than managing people, and you'll split your time between reviewing evals, designing new ones, and building the agent tooling that scales your own judgment.

We're an early-stage startup, so you should expect and enjoy that your responsibilities will grow and priorities change quickly.

About us

Safety data needs to keep pace with capabilities. As AI advances, we build evaluations that measure consequential agent failures and produce training data that helps mitigate them.

Although we're only a year old, we've already been featured in Anthropic's Fable 5 & Mythos 5 System Card, led Anthropic's prompt injection evaluation of Auto mode, and been featured in Meta's Muse Spark Safety & Preparedness Report.

Our areas of focus include red teaming, control, anti-distillation, AI safety reasoning, reward-hacking propensity evaluations, and embedded assessments.

A lot of companies say they use AI extensively. We mean it. PR count isn't the goal, but to give a concrete example, last week a single engineer merged 1,000 PRs.

About the role

Trajectory collects data on model failures from our red-teaming. We also work with AI safety researchers to collect data on how models can help with their research. You will:

1. Create evals to make sure that the data we collect are robust and fit our customers’ needs.

2. Turn our data into evals of our customers’ models.

3. Set evals strategy. Decide what evals Trajectory needs to build next to scale our data collection, test the safety of models, and communicate findings to frontier labs and the public.

Your first project will be to build evals to make sure our red-teamers’ prompt injection attacks are robust. To do that, you'll:

• Hold the quality bar. Review tasks, transcripts, and red-teamer submissions, and decide what ships to frontier lab customers.

• Automate your own judgment. Turn your review patterns into agent skills and pipeline automation.

• Figure out what to test next. Study why models succeed and why they fail, then pick task types and attack methods to find new failures.

• Improve how we work. Build evals for our internal agentic pipelines to help us scale rapidly while maintaining our quality bar.

Future projects may include:

• Evaluating data quality for red-teaming beyond prompt injection, e.g., red-teaming of eval awareness and reward hacking.

• Building evals to assess our customers’ models on conceptual reasoning and automated alignment research.

You'll work closely with our founders and the teams developing frontier models, with unusual autonomy to make consequential decisions. The work you review shapes system cards, deployment safeguards, and how much the world can trust the most capable AI systems.

What we're looking for

Essential

• Calibrated eval judgment: you can tell when a task, transcript, grade, or environment is subtly wrong, and explain the evidence behind your call.

• Agent-native engineering: you use LLMs and coding agents as core tools, and you can independently automate your own workflows. You have at least 2 years of experience as a software engineer (or equivalent experience).

• Sustained attention to detail: you hold the same bar on the hundredth review as on the first, and you look for ways to automate the repeatable parts.

• Early-stage startup drive: you enjoy a fast pace, shifting priorities, limited structure, and taking on whatever will have the biggest impact.

Desirable

• You’ve built an eval, benchmark, grading pipeline, or quality assurance system. Experience with agentic evals, RL environments, model training data, or LLM judges for rubric-based grading is a plus.

• Prompt injection, red teaming, or security experience

• Familiarity with AI safety and the alignment research community

• Experience collaborating with frontier labs or other demanding technical customers

Logistics

Total compensation: Includes base salary, bonus, and the estimated annualized value of equity. Your level depends on the scope, autonomy, and impact we expect you to have:

• L4: $350,000

• L5: $520,000

• L6: $670,000

• L7: $900,000

Base salary: $250,000–$400,000 per year.

Location and workspace: Our team works out of Constellation in Berkeley, CA, and we prefer someone who can work alongside us there. We are open to remote for the right candidate.

Benefits:

• Health coverage: You'll receive a generous monthly pre-tax allowance to choose the medical, dental, and vision coverage that best fits your needs.

• 401(k): We offer a 401(k) retirement plan with a company match.

• Food: Catered lunch and dinner, unlimited snacks and drinks.

• Relocation: A generous moving allowance to relocate to Berkeley.

Visa sponsorship: We sponsor visas, although we cannot successfully sponsor every visa for every role or candidate. If we make you an offer, we will make every reasonable effort to secure the visa you need, with support from an immigration lawyer we retain to guide and coordinate the process.

Reports to: Ben West, CTO.