We're looking for experienced domain experts to design very hard, computer-solvable problems that today’s leading AI systems can't crack. Each problem you write becomes a benchmark task used to measure and improve frontier AI models. Your field knowledge is what matters here. This is not a software engineering role, and you won't write code files: once you submit, a technical reviewer turns your problem into the files needed to run it.
What You’ll Do
- Design original, PhD-level problems within an assigned domain (and subdomain, where given), or in a field of your choice when none is assigned
- Make sure each problem can only be solved computationally, by running commands or scripts on a Linux machine, and not by hand or with a calculator
- Define one clear, objectively correct answer that a program can mark right or wrong, with no human judgment needed
- Make problems robust enough that two independent experts would arrive at the same answer
- Test problems against current state-of-the-art AI tools and refine them until those tools fail
- Work with reviewers to clarify problem statements, edge cases and the expected answer format
What You Bring
- 8+ years of professional or research experience in your domain
- PhD-level depth in your field (a PhD is strongly preferred; equivalent senior research or industry expertise is considered)
- Hands-on experience solving problems in your field with computational tools: simulations, numerical methods, data analysis, scripting or domain-specific software
- Comfort working in a Linux environment at a practical level
- A rigorous, precise approach: you can tell the difference between a hard problem and an ambiguous one
- Strong written English for clear, unambiguous problem statements
Nice to Have
- Experience building exams, qualifying problems, competition problems or research benchmarks
- Prior work in AI training, model evaluation or red-teaming
- Peer-reviewed publications or other recognized contributions in your field
- Familiarity with how large language models and AI agents fail on technical tasks
Why This Role
- Shape how the world’s most advanced AI systems are measured
- Fully remote and flexible: work on your own schedule
- Use deep expertise on intellectually demanding, original problems
- Join a vetted network of senior experts across scientific and technical fields