Remote LLM Benchmark Scientist - Evaluations & Validation

Anyone AI; Lima Metropolitana; On-site; PEN 407,331 - 644,941; Full time

Job summary

Anyone AI Labs is seeking a Research Scientist to own evaluation design for frontier models. You’ll design benchmarks across reasoning, coding, agents, tool-use and multi‑modal tasks, grounded in expert-verified truth and validated against multiple models.

You’ll QC results to survive buyer-side review and push for durable, informative evals as models improve. You’ll lead a team to recruit experts across coding and STEM, translate lab goals into robust evaluation pipelines, and publish public

Qualifications

• Research background in ML evaluation or benchmarking (a track record of published or open benchmarks, eval/measurement research, or equivalent hands-on work that labs have relied on).
• Deep LLM/frontier-model benchmarking expertise, with real strength in code-model and agentic evaluation.
• Fluency with the measurement problem itself: construct validity, psychometrics, rubrics, pass rates, headroom, contamination, and what makes a task genuinely discriminate a model.
• Interest in the safety side of evaluation, capability elicitation, robustness, and measuring the things that are hardest to measure honestly.
• Proven ability to hold a team or expert pool to a rigorous standard.
• Comfort with the full research loop: framing the question, running the study, and writing it up.
• Fluent English; Spanish a plus.

Responsibilities

• Evaluation research. Turn eval targets into original benchmark designs. Own the hard measurement questions: construct validity, item discrimination, headroom, reliability, contamination, and capability elicitation. Push toward evals that stay informative as models improve.
• Benchmark development. Build evaluation packages with subject‑matter experts, each with expert‑verified ground truth, multi‑model headroom results, and rigorous QC (calibration layers, severity‑weighted rubrics, deterministic verifiers).
• Experts. Recruit, calibrate, and review a pool across coding, agentic/tool‑use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty.
• Lab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design.
• Delivery & dissemination. Turn lab requests into winning sample packages and own pilots end to end. Where the work generalizes, help turn it into public benchmarks and papers: we support publishing at venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.

Skills

LLM evaluation

Benchmarking

Research

English fluency

Spanish beneficial

Education

PhD in ML or related

Tools

Python

PyTorch

JAX

Job description

Anyone AI Labs is seeking a Research Scientist to own evaluation design for frontier models. You’ll design benchmarks across reasoning, coding, agents, tool-use and multi‑modal tasks, grounded in expert-verified truth and validated against multiple models.

You’ll QC results to survive buyer-side review and push for durable, informative evals as models improve. You’ll lead a team to recruit experts across coding and STEM, translate lab goals into robust evaluation pipelines, and publish public