LLM Benchmarking Scientist — Evaluation & Safety

Anyone AI Labs; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Anyone AI Labs is seeking a researcher to advance evaluation design and benchmarking for language and agentic models. You will guide measurement development, recruit and calibrate expert reviewers, and translate findings into public benchmarks and papers for top venues.

The role emphasizes rigorous psychometrics, robust evaluation frameworks, and collaboration with AI labs to translate measurement needs into concrete evaluation designs. Proficiency in English is required, with Spanish as a plus.

Qualifications

• Proven research background in ML evaluation or benchmarking.
• Deep expertise in LLM/frontier-model benchmarking, especially code-model and agentic evaluation.
• Strong understanding of psychometrics, rubrics, and the measurement problem.
• Interest in AI safety, capability elicitation, and robustness.
• Ability to manage an expert pool and maintain rigorous standards.
• Experience with the full research loop from framing to publication.
• Fluency in English; Spanish is a plus.

Responsibilities

• Conduct evaluation research to turn eval targets into original benchmark designs.
• Own measurement questions including construct validity, item discrimination, and contamination.
• Build evaluation packages with expert-verified ground truth and rigorous QC.
• Recruit, calibrate, and review a pool of subject-matter experts across coding, agentic, and STEM fields.
• Act as a technical point of contact for AI labs to translate measurement needs into evaluation designs.
• Turn research findings into public benchmarks and papers for venues like NeurIPS, ICLR, and ACL.

Skills

ML evaluation

Benchmarking

Frontier-model benchmarking

Psychometrics

Measurement design

English fluency

Expert pool management

Job description

Anyone AI Labs is seeking a researcher to advance evaluation design and benchmarking for language and agentic models. You will guide measurement development, recruit and calibrate expert reviewers, and translate findings into public benchmarks and papers for top venues.

The role emphasizes rigorous psychometrics, robust evaluation frameworks, and collaboration with AI labs to translate measurement needs into concrete evaluation designs. Proficiency in English is required, with Spanish as a plus.