Research Engineer – AI Evaluation & Metrics

EngineersOfAI; San Francisco, Northern (CA, KY); Hybrid; USD 180,000 - 250,000; Full time

Job summary

Anthropic is hiring a Research Engineer on Takeoff Intel to design and run evaluation and measurement instruments at scale. You will build data pipelines that convert model outputs and telemetry into reliable metrics, prototype fast, and validate approaches before deployment.

You’ll work with research scientists and partner teams to define what’s worth measuring, review AI-written code, and contribute to internal and public reporting.

Qualifications

• Design, build, and run evaluation and measurement instruments at scale.
• Prototype fast and validate instruments with rapid iteration.
• Handle messy, large-volume data without over-engineering.
• Run experiments on large language models beyond mere data movement.

Responsibilities

• Design, build, and run capability evaluations and measurement instruments at scale
• Build data and analysis pipelines turning model outputs into reliable metrics
• Prototype new instruments quickly, validate, decide what to keep
• Review and supervise AI-written code as part of typical workflow
• Collaborate with research scientists and partner teams to define what’s worth measuring
• Contribute to internal write-ups and public reporting

Skills

End-to-end evaluation

Large-scale data handling

LLM experimentation

Code review & collaboration

Job description

Anthropic is hiring a Research Engineer on Takeoff Intel to design and run evaluation and measurement instruments at scale. You will build data pipelines that convert model outputs and telemetry into reliable metrics, prototype fast, and validate approaches before deployment.

You’ll work with research scientists and partner teams to define what’s worth measuring, review AI-written code, and contribute to internal and public reporting.