Remote LLM Evaluation Scientist - Benchmark Frontiers

Anyone AI; Paris; Sur place; EUR 90 000 - 140 000; Plein temps

Résumé du poste

Anyone AI Labs is seeking a dedicated Research Scientist to advance how frontier models are evaluated. You will design benchmarks, validate methodologies, and build evaluation packages across reasoning, coding, agents, and multi-modal systems.

You will lead expert recruitment, coordinate with labs and CEOs, and push for public benchmarks and papers at venues like NeurIPS and ICLR. Fluent English required; Spanish is a plus.

Qualifications

• Experience in ML evaluation or benchmarking research, with a track record of open benchmarks or eval studies.
• Deep expertise in LLM/frontier-model benchmarking, including code-model and agentic evaluation.
• Fluency with measurement concepts: construct validity, psychometrics, rubrics, headroom, contamination.
• Interest in safety-oriented evaluation, robustness, and measuring hard-to-measure capabilities.
• Ability to lead a team or expert pool to high standards; strong written and oral communication.

Responsabilités

• Evaluation research: define targets, design benchmarks, ensure validity and reliability.
• Benchmark development: create packages with expert-verified ground truth and multi-model headroom.
• Recruit, calibrate, and review a pool of experts in coding, agents, and STEM/reasoning.
• Serve as technical point of contact for labs, with CEO support, translating needs into designs.
• Deliver and disseminate: pilot projects, public benchmarks, and papers for venues like NeurIPS, ICLR, ACL.

Connaissances

ML evaluation

Benchmarking

LLM evaluation

Scientific research

English fluency

Spanish language

Description du poste

Anyone AI Labs is seeking a dedicated Research Scientist to advance how frontier models are evaluated. You will design benchmarks, validate methodologies, and build evaluation packages across reasoning, coding, agents, and multi-modal systems.

You will lead expert recruitment, coordinate with labs and CEOs, and push for public benchmarks and papers at venues like NeurIPS and ICLR. Fluent English required; Spanish is a plus.