AI Research Scientist - Frontier AI Peer Review

OxSci; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Founding equity

Direct line to founders

Open research questions

Generous token budget

Job summary

OxSci in London seeks a PhD-level researcher to lead AI-reviewer evaluation, shaping how AI reviewers are judged in science.

You will design meta-evaluations, run large-scale expert-annotation studies, and build a living taxonomy of AI-reviewer failure modes. Join a founding team with equity, flexible hours, and open research questions as you push the frontier of AI-enabled peer review.

Qualifications

• PhD or near completion in CS/ML/NLP or related field.
• Strong Python and hands-on experience with LLM evaluation and tooling.
• Experience designing expert-annotation studies and benchmarks.
• Fluency in evaluation methodology and statistics.

Responsibilities

• Own the research agenda on AI-reviewer evaluation.
• Design meta-evaluations exposing weaknesses beyond verdicts.
• Run expert-annotation studies at scale with robust protocols.
• Build a living taxonomy of AI-reviewer failure modes.
• Calibrate quality scoring by fusing human and AI judgments.
• Translate findings into concrete improvements to review agents.

Skills

Python

LLM evaluation

Evaluation methodology

Inter-annotator agreement

RAG pipelines

Education

PhD in CS/ML/NLP or related field

Job description

OxSci in London seeks a PhD-level researcher to lead AI-reviewer evaluation, shaping how AI reviewers are judged in science.

You will design meta-evaluations, run large-scale expert-annotation studies, and build a living taxonomy of AI-reviewer failure modes. Join a founding team with equity, flexible hours, and open research questions as you push the frontier of AI-enabled peer review.