Production ML Eval Engineer - Post-Training

Anthropic Limited; Seattle (WA); Hybrid; USD 500,000 - 850,000; Full time

Job summary

Anthropic is seeking a lead scientist/engineer to define and run the evaluation framework for production Claude models. You will determine which measurements matter, monitor live runs, and partner with research teams to set standards for trustworthy evals.

You will be hands-on with the eval fleet, build dashboards and reports, and influence post-training decisions. This role combines ML evaluation work with cross-functional collaboration in a hybrid office setting, Seattle area preferred.

Qualifications

• Designed, ran, and analyzed evaluations for ML models at scale.
• Strong Python skills and production systems experience.
• Ability to translate results into clear, actionable recommendations.

Responsibilities

• Steer the eval strategy for production Claude models: study and optimize the eval mix so it gives the most accurate and complete assessment of model quality
• Run and monitor the eval fleet live to understand how each Claude model is developing as it trains
• Advise teams across post-training on eval methodology, and help eval authors bring new evals up to the bar for production
• Investigate regressions in production runs and inform training interventions when appropriate
• Build the dashboards, alerts, and reports that researchers and leadership use to track model quality

Skills

Python

ML evaluation

Data analysis

System monitoring

Cross-team collaboration

Education

Bachelor’s degree or equivalent

Field relevant to the role

Job description

Anthropic is seeking a lead scientist/engineer to define and run the evaluation framework for production Claude models. You will determine which measurements matter, monitor live runs, and partner with research teams to set standards for trustworthy evals.

You will be hands-on with the eval fleet, build dashboards and reports, and influence post-training decisions. This role combines ML evaluation work with cross-functional collaboration in a hybrid office setting, Seattle area preferred.