AI Evaluation Researcher — Benchmarks, Signals & Usability

Bison; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health insurance

Dental and vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines in San Francisco, CA, is seeking a researcher to build reliable evaluation signals and research benchmarks for post-training models. You will design internal evaluations, usability studies, and auditing methods to improve data and training signals across research teams.

The role emphasizes collaboration with researchers and engineers, with opportunities to focus on one area or multiple evaluation challenges, including agentic environments and personalized behavior analysis.

Qualifications

• Bachelor’s degree or equivalent experience in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding.
• Experience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or other measurement systems.
• Strong written and verbal communication skills, with the ability to collaborate effectively across research and engineering teams.

Responsibilities

• Create internal evaluations and research signals for capabilities and behaviors important to model research and post-training.
• Develop usability evaluations that measure usefulness in real research and product workflows, and partner with the data flywheel to turn evaluation insights into better data and training signals.
• Improve evaluation robustness, including grader reliability, ambiguous ground truth, evaluator disagreement, false positives and negatives, and gaps between measured and intended behavior.
• Build benchmark auditing methodologies that help researchers understand, trust, and appropriately use evaluation signals.
• Develop specialized agentic evaluation environments and user simulators, including supporting harness development and studying cross‑user, cross‑harness and cross‑environment generalization.
• Develop evaluations for personalized preferences, biases, values, and other nuanced dimensions of model behavior in collaboration with post‑training crafting.

Skills

Evaluations design

Benchmarks

Datasets

Graders

Python

PyTorch

TensorFlow

JAX

Education

Bachelor's degree or equivalent in CS/ML/related

PhD in CS/ML or equivalent industry research

Tools

Python

Job description

Thinking Machines in San Francisco, CA, is seeking a researcher to build reliable evaluation signals and research benchmarks for post-training models. You will design internal evaluations, usability studies, and auditing methods to improve data and training signals across research teams.

The role emphasizes collaboration with researchers and engineers, with opportunities to focus on one area or multiple evaluation challenges, including agentic environments and personalized behavior analysis.