Research Scientist (Remote/US/LATAM)

Anyone AI Inc.; Polska; Hybrid; PLN 469,000 - 704,000; Full time

Job summary

Anyone AI Inc. seeks a Research Scientist for LLM Evaluations and Benchmarking to design, implement and defend frontier evaluation methodologies. You will build evaluation packages across reasoning, coding, agents, tool use and multi-modal capabilities grounded in expert-verified truth and cross-model validation.

You will own the study lifecycle from framing questions to publishing results, collaborating with labs, with CEO support. Fluency in English is required; Spanish is a plus.

Qualifications

• Experience in ML evaluation or benchmarking with published or open benchmarks.
• Deep expertise in LLM/frontier-model benchmarking, especially code-model and agentic tests.
• Strong grasp of measurement concepts: construct validity, rubrics, headroom and contamination.
• Interest in safety, capability elicitation, robustness and measuring hard-to-quantify aspects.
• Ability to drive a team or expert pool to high standards.
• Comfort with framing questions, running studies, and writing results in English.

Responsibilities

• Own evaluation research: design original benchmarks and ensure validity and reliability.
• Develop evaluation packages with expert ground truth and multi-model headroom results.
• Recruit and calibrate experts across coding, tool-use, and reasoning domains.
• Serve as technical liaison to labs with CEO backing and translate needs into designs.
• Turn lab requests into pilots and contribute to public benchmarks and papers.

Skills

ML evaluation

Benchmarking

Frontier model testing

Code-model evaluation

Agentic evaluation

Measurement theory

English fluency

Tools

Ground truth verification

Deterministic verifiers

Calibration layers

Severity-weighted rubrics

Headroom analysis

Multi-model benchmarking

Job description

Research Scientist, LLM Evaluations & Benchmarking

Anyone AI Labs
Reports to: CEO · Remote / LatAm / US

The role Evaluation is one of the hardest open problems in AI: we still don't have reliable ways to measure what frontier models can and can't do, and the field mostly runs on benchmarks that are saturated, contaminated, or measuring the wrong thing. You'll own that problem at Anyone AI, measuring frontier model capability.

This is a research role at heart: you decide what a good evaluation is , design the benchmarks that prove it, and defend the methodology under lab scrutiny. You'll build frontier-grade evaluation packages across reasoning, coding, agents, tool use, and multi-modal — grounded in expert-verified truth, validated against multiple models, and QC'd to survive buyer-side review.

Responsibilities

• Evaluation research. Turn eval targets into original benchmark designs. Own the hard measurement questions: construct validity, item discrimination, headroom, reliability, contamination, and capability elicitation. Push toward evals that stay informative as models improve.
• Benchmark development. Build evaluation packages with subject‑matter experts, each with expert‑verified ground truth, multi‑model headroom results, and rigorous QC (calibration layers, severity‑weighted rubrics, deterministic verifiers).
• Experts. Recruit, calibrate, and review a pool across coding, agentic/tool‑use, and STEM/reasoning. Be the final arbiter of correctness and frontier difficulty.
• Lab relationships. Be a technical point of contact for labs, with CEO support. Understand what they're trying to measure and translate it into an evaluation design.
• Delivery & dissemination. Turn lab requests into winning sample packages and own pilots end to end. Where the work generalizes, help turn it into public benchmarks and papers: we support publishing at venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.

What we're looking for

• Research background in ML evaluation or benchmarking (a track record of published or open benchmarks, eval/measurement research, or equivalent hands‑on work that labs have relied on).
• Deep LLM/frontier-model benchmarking expertise, with real strength in code-model and agentic evaluation.
• Fluency with the measurement problem itself: construct validity, psychometrics, rubrics, pass rates, headroom, contamination, and what makes a task genuinely discriminate a model.
• Interest in the safety side of evaluation, capability elicitation, robustness, and measuring the things that are hardest to measure honestly.
• Proven ability to hold a team or expert pool to a rigorous standard.
• Comfort with the full research loop: framing the question, running the study, and writing it up.
• Fluent English; Spanish a plus.