Remote LLM Evaluation Scientist: Benchmarking Models

Anyone AI; United States; Remote; MXN 2,626,510 - 3,677,114; Full time

Job summary

Anyone AI Labs is seeking a Research Scientist for LLM Evaluations and Benchmarking. This remote role spans LatAm/US and requires designing robust evaluation methodologies for frontier models, building benchmarks across reasoning, coding, agents, tool use, and multi-modal tasks.

The candidate will lead expert pools, validate ground truth, and publish results in venues like NeurIPS, ICLR, and ACL, with strong English proficiency and preference for Spanish speakers.

Qualifications

• Experience in ML evaluation or benchmarking with published benchmarks.
• Deep knowledge of LLM/frontier-model benchmarking, including code and agentic eval.
• Understanding of construct validity, psychometrics, rubrics, headroom.
• Interest in safety and robustness in evaluation.
• Ability to lead a team of experts and run full research cycles.

Responsibilities

• Research evaluation methods and design new benchmarks.
• Develop evaluation packages with ground truth and multi-model headroom.
• Recruit and calibrate a pool of experts across coding, agents, STEM.
• Serve as technical point of contact for labs with CEO support.
• Turn lab requests into pilot studies and publish benchmarks/papers.

Skills

ML evaluation

Benchmarking

Frontier-model benchmarking

Code-model evaluation

Agentic evaluation

Measurement theory

English fluency

Job description

Anyone AI Labs is seeking a Research Scientist for LLM Evaluations and Benchmarking. This remote role spans LatAm/US and requires designing robust evaluation methodologies for frontier models, building benchmarks across reasoning, coding, agents, tool use, and multi-modal tasks.

The candidate will lead expert pools, validate ground truth, and publish results in venues like NeurIPS, ICLR, and ACL, with strong English proficiency and preference for Spanish speakers.