LLM Benchmarking Research Scientist

Anyone AI Inc.; Northern (KY); Hybrid; USD 110,000 - 160,000; Full time

Job summary

Anyone AI Labs in a remote capacity across LatAm/US seeks a Research Scientist focused on LLM evaluations and benchmarking. You will design frontier-grade evaluation methods and build benchmarks across reasoning, coding, agents, and multi-modal capabilities.

You will collaborate with labs, defend methodologies under scrutiny, and contribute to public benchmarks and papers, targeting venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.

Qualifications

• Research background in ML evaluation or benchmarking.
• Expertise in LLM/frontier-model benchmarking.
• Understanding measurement constructs and psychometrics.
• Interest in safety and robustness of evaluation.
• Ability to lead a team and maintain high standards.
• Experience framing, running studies, and writing results.

Responsibilities

• Evaluation research: convert targets into benchmark designs.
• Benchmark development with expert-verified ground truth.
• Recruit and calibrate a pool of experts across coding, agents, and reasoning.
• Serve as the technical contact for labs with CEO support.
• Turn lab requests into pilots and publish benchmarks.

Skills

ML evaluation

LLM benchmarking

Measurement science

Safety in evaluation

Team leadership

Study design

Job description

Anyone AI Labs in a remote capacity across LatAm/US seeks a Research Scientist focused on LLM evaluations and benchmarking. You will design frontier-grade evaluation methods and build benchmarks across reasoning, coding, agents, and multi-modal capabilities.

You will collaborate with labs, defend methodologies under scrutiny, and contribute to public benchmarks and papers, targeting venues like NeurIPS Datasets & Benchmarks, ICLR, and ACL.