Research, Post-Training Evals

Thinking Machines Lab Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 140,000 - 200,000; Full time

Job summary

Thinking Machines Lab Inc. is seeking a researcher to help develop reliable model evaluations for research signals, spanning evaluation creation, usability, auditing, and efficiency.

You will work with researchers and engineers across post-training and the broader research org. Depending on interests, you may focus on one area or work across several problems.

Qualifications

• Bachelor’s degree or equivalent in CS/ML/Physics/Math with strong theory and empiricism.
• Experience designing or analyzing evaluations, benchmarks, datasets, graders, or measurement systems.
• Strong written and verbal communication across research and engineering teams.

Responsibilities

• Create internal evaluations and research signals for capabilities and behaviors in model research.
• Develop usability evaluations to measure usefulness in research and product workflows.
• Improve evaluation robustness including grader reliability and ground truth ambiguity.
• Build benchmark auditing methodologies for trust in signals.
• Develop agentic evaluation environments and user simulators.
• Collaborate on personalized preferences, biases, and other nuanced model behaviors.

Skills

LLMs

Post-training evaluation

Reinforcement learning

Agentic systems

Python

Distributed training debugging

Education

Bachelors in CS/ML/Physics/Math

PhD in CS/ML/Physics/Math or equivalent

Tools

PyTorch

TensorFlow

JAX

Job description

About Thinking Machines

The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.

About the Role

We’re looking for a researcher to help develop reliable model evaluations for research signals. This role spans evaluation creation, usability, auditing, and efficiency.

You’ll work closely with researchers and engineers across post-training and the broader research organization. Depending on your interests and experience, you may focus on one area or work across several of these problems.

What You’ll Do

• Create internal evaluations and research signals for capabilities and behaviors important to model research and post-training.

• Develop usability evaluations that measure whether models are genuinely useful in real research and product workflows, and partner with the data flywheel to turn evaluation insights into better data and training signals.

• Improve evaluation robustness, including grader reliability, ambiguous ground truth, evaluator disagreement, false positives and negatives, and gaps between measured and intended behavior.

• Build benchmark auditing methodologies that help researchers understand, trust, and appropriately use evaluation signals.

• Develop specialized agentic evaluation environments and user simulators, including supporting harness development and studying cross-user, cross-harness and cross-environment generalization.

• Develop evaluations for personalized preferences, biases, values, and other nuanced dimensions of model behavior in collaboration with post-training crafting.

Skills and Qualifications

Minimum qualifications:

• Bachelor’s degree or equivalent experience in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding.

• Experience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or other measurement systems.

• Strong written and verbal communication skills, with the ability to collaborate effectively across research and engineering teams.

Preferred qualifications — we encourage you to apply if you meet some but not all of these:

• Experience with LLMs, post-training, reinforcement learning, or agentic systems.

• Experience with evaluation auditing, human evaluations, LLM-judges, or open-ended task evaluation.

• Experience with agentic evaluation, harnesses, long-horizon tasks, or RL environments.

• Experience evaluating preferences, personalization, biases, values, or other nuanced model behaviors.

• Track record of developing new evaluation methodologies or research signals that meaningfully influenced model development.

• Proficiency in Python and familiarity with at least one deep learning framework (e.g., PyTorch, TensorFlow, or JAX). Comfortable with debugging distributed training and writing code that scales.

• Strong research judgment: clean ablations, honest baselines, and clear technical writing.

• PhD in Computer Science, Machine Learning, Physics, Mathematics, or a related discipline with strong theoretical and empirical grounding; or, equivalent industry research experience.