Lead AI Red Teaming & Evaluation Engineer

Trajectory Labs, PBC; Berkeley (CA); Hybrid; USD 200,000 - 400,000; Full time

Benefits offered by this job

Health coverage stipend

401(k)

Visa sponsorship

Equity

Job summary

Trajectory Labs, PBC is seeking a senior individual-contributor to own the evaluation pipeline for our prompt injection red-teaming line. You will review evals, design new tasks, and build agent tooling that scales your judgment, while working closely with founders and frontier labs.

As an early-stage startup, expect evolving priorities and significant autonomy to shape the role. Location is Berkeley, CA, with openness to remote for the right candidate and a compensation range of

Qualifications

• Calibrated eval judgment to identify subtle errors and support evidence-based decisions.
• Agent-native engineering using LLMs and coding agents to automate workflows.
• Strong attention to detail across repeated reviews and automation opportunities.
• Thrives in fast-moving, uncertain startup environments.
• Proven experience evaluating or owning an eval or grading pipeline for technical products.
• Experience building LLM judges and rubric-based grading, adapting as models change.
• Designing eval environments and tooling for evaluation tasks.
• Expertise in prompt injection, red teaming, or security in eval design.
• Familiarity with AI safety and the alignment research community.
• Experience collaborating with frontier labs or demanding technical customers.

Responsibilities

• Hold the quality bar by reviewing tasks, transcripts, and submissions and deciding what ships to customers.
• Design the next tests by analyzing model weaknesses and creating targeted tasks and environments.
• Automate judgment by turning review patterns into agent skills and pipelines.
• Improve internal workflows by applying eval principles to tooling and processes.

Skills

Calibrated eval judgment

Agent-native engineering

Sustained attention to detail

Early-stage startup drive

Rigorous evaluation experience

LLM judges & rubric-based grading

Evaluation environments

Prompt injection & red teaming

AI safety familiarity

Collaboration with frontier labs

Job description

Overview

You'll build the evaluation pipeline that checks attacks found through our prompt injection red-teaming, and you’ll decide which attacks ship to frontier lab customers.

This is a senior individual-contributor role. You'll mostly be directing agents rather than managing people, and you'll split your time between reviewing evals, designing new ones, and building the agent tooling that scales your own judgment.

We're an early-stage startup, so you should expect and enjoy that your responsibilities will grow and priorities change quickly.

About us

Safety data needs to keep pace with capabilities. As AI advances, we build evaluations that measure consequential agent failures and produce training data that helps mitigate them.

Although we're only a year old, we've already been featured in Anthropic's Fable 5 & Mythos 5 System Card, led Anthropic's prompt injection evaluation of Auto mode, and been featured in Meta's Muse Spark Safety & Preparedness Report.

Our areas of focus include red teaming, control, anti-distillation, AI safety reasoning, reward-hacking propensity evaluations, and embedded assessments.

A lot of companies say they use AI extensively. We mean it. PR count isn't the goal, but to give a concrete example, last week a single engineer merged 1,000 PRs.

About the role

Frontier labs use our data to make their models robust to prompt injection. That only works if the data are right: a subtly broken task or a wrong grade teaches the model the wrong lesson.

You'll:

• Hold the quality bar. Review tasks, transcripts, and red-teamer submissions, and decide what ships to frontier lab customers.

• Automate your own judgment. Turn your review patterns into agent skills and pipeline automation.

• Figure out what to test next. Study why models succeed and why they fail, then pick task types and attack methods to find new failures.

• Improve how we work. Build evals for our internal agentic pipelines to help us scale rapidly while maintaining our quality bar.

You'll work closely with our founders and the teams developing frontier models, with unusual autonomy to make consequential decisions. The work you review shapes system cards, deployment safeguards, and how much the world can trust the most capable AI systems.

Your primary focus will be in our AI Red Teaming workstream. However, we build evals across several areas, and your responsibilities may expand over time.

What we're looking for

Essential

• Calibrated eval judgment: you can tell when a task, transcript, grade, or environment is subtly wrong, and explain the evidence behind your call.

• Agent-native engineering: you use LLMs and coding agents as core tools, and you can independently automate your own workflows. You have at least 2 years of experience as a software engineer (or equivalent experience).

• Sustained attention to detail: you hold the same bar on the hundredth review as on the first, and you look for ways to automate the repeatable parts.

• Early-stage startup drive: you enjoy a fast pace, shifting priorities, limited structure, and taking on whatever will have the biggest impact.

Desirable

• You’ve built an eval, benchmark, grading pipeline, or quality assurance system. Experience with agentic evals, RL environments, model training data, or LLM judges for rubric-based grading is a plus.

• Prompt injection, red teaming, or security experience

• Familiarity with AI safety and the alignment research community

• Experience collaborating with frontier labs or other demanding technical customers

We encourage you to apply if you think you might be a good fit, even if you’re not sure that you meet all of the requirements.

We’re also hiring for Head of Evals. Apply to whichever role seems like the better fit; we can discuss both during the hiring process.

Role details and benefits

• Reports to: Ben West, CTO.

• Location and workspace: Our team works out of Constellation in Berkeley, CA, and we prefer someone who can work alongside us there. We are open to remote for the right candidate.

• Compensation: $200,000–$325,000 plus equity.

• Health coverage: You'll receive a generous monthly pre-tax allowance to choose the medical, dental, and vision coverage that best fits your needs.

• 401(k): We offer a 401(k) retirement plan with a company match.

• Equity donation matching: Donate up to 100% of your equity to a 501(c)(3) charity, and we'll match it.

• Food: Lunch, dinner, and unlimited snacks and drinks at Constellation, so you never lose time to cooking.

• Relocation: A generous moving allowance to relocate to Berkeley.

• Visa sponsorship: We sponsor visas, although we cannot successfully sponsor every visa for every role or candidate. If we make you an offer, we will make every reasonable effort to secure the visa you need, with support from an immigration lawyer we retain to guide and coordinate the process.