Evaluation Platform Engineer: Build Scalable ML Benchmarks

Thinking Machines Lab; San Francisco (CA); On-site; USD 300,000 - 475,000; Full time

Benefits offered by this job

Health, dental, and vision benefits

Unlimited PTO

Paid parental leave

Relocation support

Job summary

Thinking Machines, based in San Francisco, seeks an experienced engineer to design and build a self-serve evaluation platform for frontier AI models. You will work across Python frameworks, data pipelines, APIs, and front-end interfaces to enable reproducible evaluations and insights that guide research and model releases.

You will collaborate with pre-training, post-training, and applied teams to improve evaluation methods and turn results into decisive research actions within a fast-paced

Qualifications

• A bachelor's degree, or equivalent practical experience, in CS, engineering, ML, or a related field.
• Two years of post-grad work as a software or ML engineer.
• Experience building evaluations, benchmarks, graders, or model-quality systems.
• Strong software engineering fundamentals for reliable, maintainable systems.
• Proficiency in Python or Rust; frontend uses React/TypeScript.
• Experience with databases, data pipelines, or distributed data infrastructures.
• Ability to own projects from discovery to deployment.
• Experience collaborating with researchers and cross-functional partners.

Responsibilities

• Design, build, and maintain evaluation platform end to end.
• Work across evaluation libraries, data pipelines, APIs, and front-end apps.
• Create abstractions for evaluation tasks, environments, datasets, and outputs.
• Ensure reproducibility with versioning, provenance and observability.
• Collaborate with researchers to streamline bespoke workflows.
• Work with internal research platform team to integrate tools.

Skills

Python

Rust

React/TypeScript

Data pipelines

Distributed systems

Backend development

Cross-functional teamwork

Education

Bachelor's degree in CS or related

Tools

PostgreSQL

Airflow

Docker

Kubernetes

Job description

Thinking Machines, based in San Francisco, seeks an experienced engineer to design and build a self-serve evaluation platform for frontier AI models. You will work across Python frameworks, data pipelines, APIs, and front-end interfaces to enable reproducible evaluations and insights that guide research and model releases.

You will collaborate with pre-training, post-training, and applied teams to improve evaluation methods and turn results into decisive research actions within a fast-paced