Research Engineer, AI Systems & Evaluation

AItoolnavio; San Francisco (CA); On-site; USD 140,000 - 210,000; Full time

Job summary

Outset, a San Francisco-based company pioneering AI-powered user interviews, seeks a Research Engineer to design and maintain scalable evaluation systems for AI models. You will support scientists, ensure reproducible experiments, and bridge research with production datasets across diverse studies.

Join a tight-knit team backed by top investors, contributing to the development of digital twins and AI-driven research methods used by global brands.

Qualifications

• Several years of software engineering experience in a lab or research setting.
• Experience translating scientific requirements into production-ready code.
• Experience building/operating large-scale data pipelines for human behavior data.

Responsibilities

• Own evaluation infrastructure for testing modeling approaches.
• Implement measurement pipelines with deterministic outputs.
• Maintain versioned evaluation datasets with held-out sets.
• Build data pipelines for human-participant data and model training data.
• Keep research code in step with production product and ship improvements.
• Detect and prevent threats to evaluation validity.
• Track compute and API spend across experiments.
• Manage repositories, code review, and reproducibility of workflows.
• Identify implementation choices for correctness and efficiency.

Skills

Python

Data pipelines

Large language models

Cloud infrastructure

Deterministic code

SQL

Git workflows

Tools

Terraform

Docker

Kubernetes

vLLM

Job description

Outset, a San Francisco-based company pioneering AI-powered user interviews, seeks a Research Engineer to design and maintain scalable evaluation systems for AI models. You will support scientists, ensure reproducible experiments, and bridge research with production datasets across diverse studies.

Join a tight-knit team backed by top investors, contributing to the development of digital twins and AI-driven research methods used by global brands.