Production ML Evaluation Engineer - Post-Training

Anthropic Limited; San Francisco (CA); On-site; USD 500,000 - 850,000; Full time

Job summary

Anthropic is seeking a lead scientist to define how we evaluate production training runs for Claude models. You will decide which measurements matter, monitor the eval fleet in live training, and set the standard for trustworthy post‑training assessment across teams.

You will collaborate with research groups across RL domains, translate findings into practical guidance for production and research, and build dashboards, alerts, and reports that researchers and leadership rely on to track model

Qualifications

• Designed, run, and analyzed evaluations for ML models at scale.
• Care deeply about measurement quality and trust in numbers.
• Translate ambiguous results into clear, actionable recommendations.
• Strong Python skills and comfort with production systems.
• Maintain clarity and rigor when debugging time-sensitive issues.

Responsibilities

• Steer the eval strategy for production Claude models and optimize the eval mix.
• Run and monitor the eval fleet during live training runs.
• Advise teams across post-training on eval methodology and improve evaluation standards.
• Investigate regressions in production runs and inform training interventions.
• Build dashboards, alerts, and reports used by researchers and leadership.

Skills

Python

ML evaluation

Data analysis

Production systems

Education

Bachelor’s degree

Job description

Anthropic is seeking a lead scientist to define how we evaluate production training runs for Claude models. You will decide which measurements matter, monitor the eval fleet in live training, and set the standard for trustworthy post‑training assessment across teams.

You will collaborate with research groups across RL domains, translate findings into practical guidance for production and research, and build dashboards, alerts, and reports that researchers and leadership rely on to track model