Frontier AI Evaluation Scientist: Research & Benchmarks

Hidden Jobs; United States; Remote; USD 250,000 - 350,000; Full time

Benefits offered by this job

Equity

Publish at top conferences

High autonomy

Job summary

Hidden Jobs seeks a researcher to advance how frontier AI systems are evaluated, trained, and improved. You will tackle benchmark gaps, generate synthetic STEM data, and rigorously evaluate model reliability and calibration.

The role emphasizes hypothesis generation, experimentation, and publication across frontier STEM evaluation and data generation. The ideal candidate holds a PhD or equivalent and demonstrates strong experimental design, quantitative analysis, and knowledge of modern LLMs.

Qualifications

• PhD or equivalent research experience in a highly technical field.
• Ability to formulate and execute original research projects.
• Strong grounding in modern large language models and frontier AI research.
• Excellent experimental design and quantitative analysis skills.

Responsibilities

• Identify important gaps in benchmark and evaluation literature and design novel benchmarks within and across STEM domains.
• Develop rigorous task-generation, grading, contamination-control, difficulty-calibration, and validation methodologies that set new standards for evaluating frontier models.
• Create methods for generating high-quality synthetic STEM training data and run experiments that separate genuine capability gains from raw training volume.
• Investigate hallucination, uncertainty, calibration, and epistemic failure, and build evaluations for factual reliability, self-correction, verification, citation, and appropriate abstention.
• Research long-horizon scientific agents and design workflows combining literature search, coding, simulation, tool use, experimentation, and iterative reasoning.
• Pursue new research directions in reasoning, model evaluation, AI-for-science, data generation, and emerging capabilities.

Skills

Original research ability

Experimental design

Quantitative analysis

Knowledge of frontier AI/LLMs

Education

PhD or equivalent

Tools

Python

Simulation tools

Job description

Hidden Jobs seeks a researcher to advance how frontier AI systems are evaluated, trained, and improved. You will tackle benchmark gaps, generate synthetic STEM data, and rigorously evaluate model reliability and calibration.

The role emphasizes hypothesis generation, experimentation, and publication across frontier STEM evaluation and data generation. The ideal candidate holds a PhD or equivalent and demonstrates strong experimental design, quantitative analysis, and knowledge of modern LLMs.