Remote LLM Evaluation Scientist - Benchmark Frontiers

Anyone AI; Paris; On-site; EUR 90,000 - 140,000; Full time

Job summary

Anyone AI Labs is seeking a dedicated Research Scientist to advance how frontier models are evaluated. You will design benchmarks, validate methodologies, and build evaluation packages across reasoning, coding, agents, and multi-modal systems.

You will lead expert recruitment, coordinate with labs and CEOs, and push for public benchmarks and papers at venues like NeurIPS and ICLR. Fluent English required; Spanish is a plus.

Qualifications

• Experience in ML evaluation or benchmarking research, with a track record of open benchmarks or eval studies.
• Deep expertise in LLM/frontier-model benchmarking, including code-model and agentic evaluation.
• Fluency with measurement concepts: construct validity, psychometrics, rubrics, headroom, contamination.
• Interest in safety-oriented evaluation, robustness, and measuring hard-to-measure capabilities.
• Ability to lead a team or expert pool to high standards; strong written and oral communication.

Responsibilities

• Evaluation research: define targets, design benchmarks, ensure validity and reliability.
• Benchmark development: create packages with expert-verified ground truth and multi-model headroom.
• Recruit, calibrate, and review a pool of experts in coding, agents, and STEM/reasoning.
• Serve as technical point of contact for labs, with CEO support, translating needs into designs.
• Deliver and disseminate: pilot projects, public benchmarks, and papers for venues like NeurIPS, ICLR, ACL.

Skills

ML evaluation

Benchmarking

LLM evaluation

Scientific research

English fluency

Spanish language

Job description

Anyone AI Labs is seeking a dedicated Research Scientist to advance how frontier models are evaluated. You will design benchmarks, validate methodologies, and build evaluation packages across reasoning, coding, agents, and multi-modal systems.

You will lead expert recruitment, coordinate with labs and CEOs, and push for public benchmarks and papers at venues like NeurIPS and ICLR. Fluent English required; Spanish is a plus.