LLM Evaluation & Benchmarking Scientist

Anyone AI; Colombia; Presencial; COP 367.462.836 - 567.897.110; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Remote work

Opportunity to publish

Descripción de la vacante

Anyone AI Labs is seeking a Research Scientist focused on LLM evaluations and benchmarking. You will design frontier-grade evaluation packages across reasoning, coding, agents, and multi-modal capabilities, grounded in expert-verified truth and validated against multiple models.

You’ll own evaluation methodology, collaborate with labs, and contribute to public benchmarks and papers, with a strong emphasis on rigor and QC in a remote, LatAm/US‑centric setup.

Formación

• Track record of published or open benchmarks, eval/measurement research, or equivalent hands-on work that labs have relied on.
• Deep LLM/frontier-model benchmarking expertise, with real strength in code‑model and agentic evaluation.
• Fluency with the measurement problem: construct validity, psychometrics, rubrics, headroom, contamination, and what makes a task discriminate a model.
• Interest in the safety side of evaluation, capability elicitation, robustness, and measuring the hardest-to-measure things.
• Proven ability to hold a team or expert pool to a rigorous standard.
• Comfort with the full research loop: framing the question, running the study, and writing it up.
• Fluent English; Spanish a plus.

Responsabilidades

• Evaluation research: design original benchmark targets and measures.
• Benchmark development with expert-verified ground truth and multi-model headroom results.
• Recruit, calibrate, and review a pool of experts across coding, agentic/tool-use, and STEM/reasoning.
• Act as technical point of contact for labs and translate needs into evaluation designs.
• Turn lab requests into pilots and publish public benchmarks and papers when applicable.

Conocimientos

ML evaluation research

Frontier-model benchmarking

Code-model evaluation

Agentic evaluation

Measurement theory

Safety in evaluation

Team leadership

English fluency

Spanish (plus)

Descripción del empleo

Anyone AI Labs is seeking a Research Scientist focused on LLM evaluations and benchmarking. You will design frontier-grade evaluation packages across reasoning, coding, agents, and multi-modal capabilities, grounded in expert-verified truth and validated against multiple models.

You’ll own evaluation methodology, collaborate with labs, and contribute to public benchmarks and papers, with a strong emphasis on rigor and QC in a remote, LatAm/US‑centric setup.