PhD Scientist — AI Benchmarking

Toptal

España

Remote

EUR 83,000 - 165,000

Part time

2 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Toptal is seeking a PhD-level scientist to craft one original benchmark problem in your domain. The task requires Linux-based scripting to reach a single, programmatically verifiable answer and a clearly specified output format.

You will write a concise pitch and a final problem statement, then red-team the problem against top AI tools, documenting failures. This remote, contract-based engagement emphasizes rigor, reproducibility, and reviewer iteration.

Qualifications

  • PhD-level scientists to design rigorous benchmark problems exposing AI gaps.
  • Ability to formulate problems that require Linux-based solutions and verifiable answers.
  • Experience writing precise problem statements with explicit file paths and outputs.

Responsibilities

  • Design one highly challenging benchmark problem in your domain.
  • Guarantee a single, unambiguous answer that is programmatically verifiable.
  • Write a pitch and final problem statement with exact paths and output format.
  • Red-team the problem against leading AI tools and document failures.
  • Deliver a complete expert solution package with steps and licenses.
  • Iterate with reviewers until acceptance.

Skills

Linux command line
Scripting
Benchmarks design
Problem formulation
Scientific rigor
AI evaluation

Education

PhD in science/engineering

Tools

Python
Docker

Job description

AI systems are getting very good at answering science questions. They are far less reliable when they have to do science: pull real data, write and run code in a Linux terminal, and arrive at a correct, checkable result. We're looking for PhD-level scientists to build the problems that expose that gap.

You’ll partner with a leading AI data company that creates expert-written benchmarks used to evaluate and improve frontier AI models. Your job is to design one original, PhD-level problem in your assigned science domain and subdomain - a problem that a terminal-based AI agent must solve with Linux commands or scripts, that has a single programmatically verifiable answer, and that today's leading AI tools fail to solve.

This is a focused, contract-based engagement where scientific rigor, reproducibility, and precise problem design matter more than volume.

What You’ll Do
  • Design one highly challenging benchmark problem in your assigned domain and subdomain that requires Linux commands or scripting to solve.
  • Guarantee a single, unambiguous answer that can be verified automatically - no judgment calls, no partial-credit interpretation.
  • Write a pitch and a final problem statement that states the goal upfront and specifies full file paths, exact output format, units, and allowed margin of error.
  • Red-team your problem against leading AI tools, and document clearly why and where the models fail.
  • Deliver a complete expert solution package: step-by-step solution, expected results, automated checks, and the source and license details for all input data.
  • Iterate with reviewers — incorporate feedback and correct AI-generated solution plans and checklists until your task is accepted.
What You Bring
  • A completed PhD from a top-500 university in a science field, or an MD with a clinical specialization.
  • Deep domain experience: 10+ years in your scientific field, or 3+ years plus at least 3 peer-reviewed publications.
  • Expertise in one science area such as physics, chemistry, biology, mathematics, medicine, earth science, or engineering.
  • Comfort working in the Linux command line and writing scripts to evaluate and verify solutions.
  • Strong written English — you can write problem statements that leave no room for misinterpretation.
  • Availability for contract-based work.
Nice to Have
  • Experience writing benchmark, evaluation, or red-teaming tasks for AI labs or AI training platforms.
  • Proficiency in Python and scientific computing libraries (e.g., NumPy, SciPy, pandas, domain-specific packages) for simulation, data analysis, or numerical work.
  • A publication record in peer-reviewed journals in your field.
  • Experience with reproducible computational workflows, containers (e.g., Docker), or scientific data pipelines.
Why This Role
  • Shape how frontier AI is measured. Your benchmark becomes part of how leading AI systems are evaluated and improved on real scientific work.
  • Use your actual expertise. The difficulty comes from the science itself - the kind of computational problem only someone with your training would know how to pose and verify.
  • Defined scope, flexible schedule. One well-scoped deliverable, worked remotely on your own time.
  • Work with expert reviewers who care about precision as much as you do.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Biology Expert - AI Trainer (Spain)
Biology Expert - AI Trainer (Spain)

Anyone AI Inc. • Spain

Remote
EUR 41,000 - 58,000
Postdoctoral Research Scientist (Remote | $60–$120/hr)
Postdoctoral Research Scientist (Remote | $60–$120/hr)

Synthires • Spain

Remote
EUR 75,000 - 150,000
Healthcare & Medical Expert
Healthcare & Medical Expert

Anyone AI • Madrid

Remote
EUR 37,000 - 74,000
Mathematics Expert (Spain)
Mathematics Expert (Spain)

Anyone AI Inc. • Spain

Remote
EUR 30,000 - 49,000
Climate Data Specialist
Climate Data Specialist

Alignerr Corp. • Spain

Remote
EUR 13,000 - 50,000
Fully remote
Contract-based freelance role
Flexible schedule 10–40 hours/week
Machine Learning — PhD Intern (Benchmarking for Inverse Design and Optimization)
Machine Learning — PhD Intern (Benchmarking for Inverse Design and Optimization)

Keysight Technologies, Inc. • Barcelona

On-site
EUR 13,000 - 20,000
Precision Medicine Data Lead
Precision Medicine Data Lead

Alignerr Corp. • Madrid

Remote
EUR 55,000 - 165,000
Precision Medicine Data Lead
Precision Medicine Data Lead

Alignerr Corp. • Spain

Remote
EUR 83,000 - 165,000
ML Research Expert — Published First Author (Benchmark Authoring)
ML Research Expert — Published First Author (Benchmark Authoring)

AfterQuery • Spain

Remote
EUR 155,000 - 221,000
Fully remote
Flexible hours
Competitive hourly compensation
Physics Expert - AI Trainer (Spain)
Physics Expert - AI Trainer (Spain)

Anyone AI Inc. • Spain

Remote
EUR 37,000 - 49,000