Staff AI Safety & Evaluation Engineer

Anthropic Limited; New York (NY); On-site; USD 320,000 - 485,000; Full time

Benefits offered by this job

Competitive compensation

Equity donation matching

Generous vacation

Parental leave

Flexible working hours

Job summary

Anthropic Limited is seeking a researcher/engineer to design experiments and build evaluation methods for safety and alignment. You will develop datasets representing real-world misuse, create agentic investigation evaluation tools, and ensure integration into training and deployment pipelines.

The role sits at the intersection of applied ML research and engineering, requiring collaboration with policy, enforcement, and training teams to drive safer model behavior and robust evaluation coverage.

Qualifications

• Experience building and maintaining data pipelines.
• Experience with LLMs and understanding of their capabilities and failure modes, especially agentic systems with tool use and multi-step reasoning.
• Strong data analysis skills and ability to draw reliable insights from large datasets.
• Ability to move between research prototyping and production-quality code.
• Ability to translate ambiguous problems into concrete, testable experiments.
• Care deeply about AI safety and want real impact.

Responsibilities

• Design and run experiments to improve evaluation quality and generate representative test data.
• Build evaluation harnesses for agentic investigation systems and define metrics and grading approaches.
• Construct high-quality eval datasets representing real-world misuse across harm areas.
• Ship research into evaluation and release pipelines that run during model training and updates.

Skills

Data pipelines

LLMs

Data analysis

Research prototyping

Production code

Education

Bachelor's degree

Tools

LLM evaluation frameworks

Evaluation tools

Job description

Anthropic Limited is seeking a researcher/engineer to design experiments and build evaluation methods for safety and alignment. You will develop datasets representing real-world misuse, create agentic investigation evaluation tools, and ensure integration into training and deployment pipelines.

The role sits at the intersection of applied ML research and engineering, requiring collaboration with policy, enforcement, and training teams to drive safer model behavior and robust evaluation coverage.