Lead Post-Training Model Evaluations Engineer

Anthropic; Seattle (WA); Hybrid; USD 500,000 - 850,000; Full time

Job summary

Anthropic, a leading AI safety and governance company, is seeking a senior role to lead the science of evaluating production training runs for Claude models. You will define measurement strategies, partner with research teams, and directly operate the eval fleet to ensure trustworthy model quality during training.

In this role, you will collaborate across RL domains, shape post-training evaluation standards, and be hands-on with live results to inform release decisions.

Qualifications

• Hands-on experience post-training large language models.
• Research experience in ML evaluation or benchmarking.
• Background in statistics and experimental design.
• Experience developing robust evaluation metrics for ML systems.

Responsibilities

• Steer the eval strategy for production Claude models: study and optimize the eval mix so it gives the most accurate and complete assessment of model quality
• Run and monitor the eval fleet live to understand how each Claude model is developing as it trains
• Advise teams across post-training on eval methodology, and help eval authors bring new evals up to the bar for production
• Investigate regressions in production runs and inform training interventions when appropriate
• Build the dashboards, alerts, and reports that researchers and leadership use to track model quality

Skills

Python

ML evaluation

Statistics

Production systems

Education

Bachelor's degree

Job description

Anthropic, a leading AI safety and governance company, is seeking a senior role to lead the science of evaluating production training runs for Claude models. You will define measurement strategies, partner with research teams, and directly operate the eval fleet to ensure trustworthy model quality during training.

In this role, you will collaborate across RL domains, shape post-training evaluation standards, and be hands-on with live results to inform release decisions.