AI systems are getting very good at answering science questions. They are far less reliable when they have to do science: pull real data, write and run code in a Linux terminal, and arrive at a correct, checkable result. We're looking for PhD-level scientists to build the problems that expose that gap.
You’ll partner with a leading AI data company that creates expert-written benchmarks used to evaluate and improve frontier AI models. Your job is to design one original, PhD-level problem in your assigned science domain and subdomain - a problem that a terminal-based AI agent must solve with Linux commands or scripts, that has a single programmatically verifiable answer, and that today's leading AI tools fail to solve.
This is a focused, contract-based engagement where scientific rigor, reproducibility, and precise problem design matter more than volume.
What You’ll Do
- Design one highly challenging benchmark problem in your assigned domain and subdomain that requires Linux commands or scripting to solve.
- Guarantee a single, unambiguous answer that can be verified automatically - no judgment calls, no partial-credit interpretation.
- Write a pitch and a final problem statement that states the goal upfront and specifies full file paths, exact output format, units, and allowed margin of error.
- Red-team your problem against leading AI tools, and document clearly why and where the models fail.
- Deliver a complete expert solution package: step-by-step solution, expected results, automated checks, and the source and license details for all input data.
- Iterate with reviewers — incorporate feedback and correct AI-generated solution plans and checklists until your task is accepted.
What You Bring
- A completed PhD from a top-500 university in a science field, or an MD with a clinical specialization.
- Deep domain experience: 10+ years in your scientific field, or 3+ years plus at least 3 peer-reviewed publications.
- Expertise in one science area such as physics, chemistry, biology, mathematics, medicine, earth science, or engineering.
- Comfort working in the Linux command line and writing scripts to evaluate and verify solutions.
- Strong written English — you can write problem statements that leave no room for misinterpretation.
- Availability for contract-based work.
Nice to Have
- Experience writing benchmark, evaluation, or red-teaming tasks for AI labs or AI training platforms.
- Proficiency in Python and scientific computing libraries (e.g., NumPy, SciPy, pandas, domain-specific packages) for simulation, data analysis, or numerical work.
- A publication record in peer-reviewed journals in your field.
- Experience with reproducible computational workflows, containers (e.g., Docker), or scientific data pipelines.
Why This Role
- Shape how frontier AI is measured. Your benchmark becomes part of how leading AI systems are evaluated and improved on real scientific work.
- Use your actual expertise. The difficulty comes from the science itself - the kind of computational problem only someone with your training would know how to pose and verify.
- Defined scope, flexible schedule. One well-scoped deliverable, worked remotely on your own time.
- Work with expert reviewers who care about precision as much as you do.