AI Evaluation Researcher: Post-Training Signals & Metrics

Mosaic.tech; San Francisco (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

Thinking Machines in San Francisco seeks a researcher to advance internal evaluations and signals for post-training models. You will collaborate with researchers and engineers across the research organization, shaping evaluation creation, usability, and auditing.

Your work will span agentic evaluations, score systems, and scalable signals that improve data and training signals, with emphasis on robustness, truth-grounding, and generalization across environments.

Qualifications

• Bachelor’s degree or equivalent in CS/ML/Physics/Math with strong grounding.
• Experience designing, building, or analyzing evaluations, benchmarks, datasets, graders, or similar systems.
• Strong written and verbal communication across research and engineering teams.

Responsibilities

• Create internal evaluations and research signals for model capabilities and behaviors.
• Develop usability evaluations for real research and product workflows.
• Improve evaluation correctness and reliability across ground truth, graders, and disagreements.
• Build onboarding and auditing methodologies for internal signals.
• Develop agentic evaluations and cross-environment generalization.
• Support scalable signal-bearing core sets for RL research.

Skills

Evaluations design

Benchmarks

Python

Deep learning frameworks

Communication

Research writing

Agentic evaluation

Harneses & cross-env

Open-ended task evaluation

Education

Bachelor’s degree in CS/ML/Physics/Math

PhD in CS/ML/Math/Physics

Tools

Python

PyTorch

TensorFlow

JAX

Job description

Thinking Machines in San Francisco seeks a researcher to advance internal evaluations and signals for post-training models. You will collaborate with researchers and engineers across the research organization, shaping evaluation creation, usability, and auditing.

Your work will span agentic evaluations, score systems, and scalable signals that improve data and training signals, with emphasis on robustness, truth-grounding, and generalization across environments.