Staff Software Engineer, AI Safety Evaluations

Anthropic Limited; New York (NY); Hybrid; USD 320,000 - 485,000; Full time

Benefits offered by this job

Competitive compensation

Equity donation matching

Generous vacation & parental leave

Flexible working hours

Office space

Job summary

Anthropic is seeking an experienced Evaluations Research Scientist to design experiments, build evaluation datasets, and ship tooling that gates model training and releases. You will work at the intersection of ML research and engineering, advancing measurement of model safety and behavior in real-world scenarios.

You will contribute to datasets representing real harm, develop agentic evaluation harnesses, and collaborate with Policy and Enforcement to translate observed patterns into measurable

Qualifications

• Experience building and maintaining data pipelines.
• Experience working with LLMs and understanding their capabilities and failure modes.
• Strong data analysis skills and ability to draw reliable insights from large datasets.
• Ability to move fluidly between research prototyping and production-quality code.
• Ability to translate ambiguous problems into concrete, testable experiments.
• Care about AI safety and want real impact.

Responsibilities

• Evaluation research and methodology: design and run experiments to improve evaluation quality and coverage.
• Agentic investigation evals: build and own evaluation harness and measure end-to-end performance.
• Datasets grounded in real harm: construct high-quality eval datasets across harm areas.
• Productionization and tooling: ship research into evaluation and release pipelines and build usable tooling.

Skills

Data pipelines

LLMs

Data analysis

Prototyping to production

Experiment design

AI safety

Education

Bachelor’s degree

Tools

Job description

Anthropic is seeking an experienced Evaluations Research Scientist to design experiments, build evaluation datasets, and ship tooling that gates model training and releases. You will work at the intersection of ML research and engineering, advancing measurement of model safety and behavior in real-world scenarios.

You will contribute to datasets representing real harm, develop agentic evaluation harnesses, and collaborate with Policy and Enforcement to translate observed patterns into measurable