Staff Software Engineer, AI Safety Evaluations

anthropic; San Francisco, Seattle (CA, WA); Hybrid; USD 320,000 - 485,000; Full time

Job summary

Anthropic Limited is seeking a Staff Software Engineer focused on Safeguards Evals. You will design experiments to evaluate model safety, build datasets representing real-world abuse, and ship evaluation tools into training and deployment pipelines.

You will own the evaluation harness for agentic investigations, measure end-to-end performance, and collaborate with policy and enforcement teams to improve model safety. This role emphasizes scalable, production-grade engineering and impact.

Qualifications

• Experience building and maintaining data pipelines.
• Experience with LLMs and understanding of agent use and failure modes.
• Strong data analysis and ability to draw insights from large data sets.
• Ability to move between research prototypes and production-grade code.
• Ability to translate ambiguous problems into concrete experiments.
• Commitment to AI safety and real-world impact.

Responsibilities

• Evaluation research and methodology: design experiments to improve evaluation quality and coverage.
• Agentic investigation evals: build evaluation harness for long-horizon agent systems and measure performance.
• Datasets grounded in real harm: create high-quality eval datasets from real traffic and synthetic data.
• Productionization and tooling: ship research into pipelines, tooling for policy teams, and scalable evaluation workflows.

Skills

Data pipelines

LLMs & agent evals

Safety & risk assessment

Production-ready code

Experiment design

Multi-turn reasoning

Education

Bachelor’s degree or equivalent

Job description

Anthropic Limited is seeking a Staff Software Engineer focused on Safeguards Evals. You will design experiments to evaluate model safety, build datasets representing real-world abuse, and ship evaluation tools into training and deployment pipelines.

You will own the evaluation harness for agentic investigations, measure end-to-end performance, and collaborate with policy and enforcement teams to improve model safety. This role emphasizes scalable, production-grade engineering and impact.