Frontier AI Evaluation Scientist (Equity + Relocation)

Cerebro; San Francisco (CA); On-site; USD 165,000 - 195,000; Full time

Benefits offered by this job

Equity

Relocation support

Health and dental insurance

Unlimited PTO

On-site benefits

Job summary

Cerebro invites applications for a Research Scientist to design novel benchmarks and evaluate frontier language models and agents. You will lead research, design experiments and collaborate with research engineers, foundation-model developers and domain experts to turn open-ended questions into scalable evaluations.

Unlike many industry roles, this position provides a direct route from research to external impact with opportunities to publish and present findings that influence how leading AI

Qualifications

• Master’s degree, PhD or equivalent research experience in ML, NLP, CS or related field.
• Strong research track record in ML, NLP, evaluation, benchmarking or an adjacent area.
• Publications at respected conferences or journals such as NeurIPS, ICML, ICLR, ACL or EMNLP.
• Excellent knowledge of experimental design, statistical analysis and research methodology.
• The ability to translate open-ended research questions into precise, testable experiments.
• Clear written and verbal communication skills.
• A highly independent approach and the ability to operate effectively within a small, research-intensive team.

Responsibilities

• Design novel benchmarks for evaluating frontier language models and agents.
• Research the limitations of existing evaluation methodologies, including benchmark contamination, model-based grading and reward hacking.
• Develop experiments that distinguish genuine capability improvements from memorisation or benchmark optimisation.
• Evaluate models across realistic, long-horizon and economically valuable tasks.
• Analyse model performance, failure modes and emergent behaviours.
• Establish statistically rigorous methods for validating benchmark quality and reliability.
• Collaborate with research engineers to implement and run evaluations at scale.
• Work with external research and industry partners to understand emerging evaluation.

Skills

Master's/PhD in ML/CS

Experimental design

Statistical analysis

Research publications

Education

Master's/PhD in ML/CS

Job description

Cerebro invites applications for a Research Scientist to design novel benchmarks and evaluate frontier language models and agents. You will lead research, design experiments and collaborate with research engineers, foundation-model developers and domain experts to turn open-ended questions into scalable evaluations.

Unlike many industry roles, this position provides a direct route from research to external impact with opportunities to publish and present findings that influence how leading AI