LLM Evaluation Scientist: Frontier Model Benchmarking

Anyone AI Inc.; Polska; Hybrid; PLN 469,000 - 704,000; Full time

Job summary

Anyone AI Inc. seeks a Research Scientist for LLM Evaluations and Benchmarking to design, implement and defend frontier evaluation methodologies. You will build evaluation packages across reasoning, coding, agents, tool use and multi-modal capabilities grounded in expert-verified truth and cross-model validation.

You will own the study lifecycle from framing questions to publishing results, collaborating with labs, with CEO support. Fluency in English is required; Spanish is a plus.

Qualifications

• Experience in ML evaluation or benchmarking with published or open benchmarks.
• Deep expertise in LLM/frontier-model benchmarking, especially code-model and agentic tests.
• Strong grasp of measurement concepts: construct validity, rubrics, headroom and contamination.
• Interest in safety, capability elicitation, robustness and measuring hard-to-quantify aspects.
• Ability to drive a team or expert pool to high standards.
• Comfort with framing questions, running studies, and writing results in English.

Responsibilities

• Own evaluation research: design original benchmarks and ensure validity and reliability.
• Develop evaluation packages with expert ground truth and multi-model headroom results.
• Recruit and calibrate experts across coding, tool-use, and reasoning domains.
• Serve as technical liaison to labs with CEO backing and translate needs into designs.
• Turn lab requests into pilots and contribute to public benchmarks and papers.

Skills

ML evaluation

Benchmarking

Frontier model testing

Code-model evaluation

Agentic evaluation

Measurement theory

English fluency

Tools

Ground truth verification

Deterministic verifiers

Calibration layers

Severity-weighted rubrics

Headroom analysis

Multi-model benchmarking

Job description

Anyone AI Inc. seeks a Research Scientist for LLM Evaluations and Benchmarking to design, implement and defend frontier evaluation methodologies. You will build evaluation packages across reasoning, coding, agents, tool use and multi-modal capabilities grounded in expert-verified truth and cross-model validation.

You will own the study lifecycle from framing questions to publishing results, collaborating with labs, with CEO support. Fluency in English is required; Spanish is a plus.