Data Flywheel Engineer - ML Evaluation & Datasets

Reflection; San Francisco (CA); On-site; USD 170,000 - 210,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave 22 weeks

Unlimited PTO

Visa sponsorship

Team events

Job summary

Reflection is a research lab focused on making intelligence open and accessible. The Data Flywheel team closes the gap between benchmark performance and real-world usefulness, building signals, data, and feedback loops to improve model behavior.

You will identify high-value data sources, onboard new data, and design evaluations that turn signals into measurable improvements across model generations. This is a hands-on, collaborative role in a fast-paced startup environment.

Qualifications

• Deep understanding of LLM training and evaluation design
• Experience building automated data/evaluation pipelines
• Strong software engineering skills for large-scale ML systems
• Track record of owning high-impact projects end-to-end

Responsibilities

• Identify high-value data sources and translate them into evaluations
• Bring data online from partner conversations to production pipelines
• Design evaluations, graders, and feedback loops for measurable model behavior
• Analyze model performance and translate insights into datasets and training signals
• Develop data strategies including vendor-based data collection
• Build infrastructure to ingest, inspect, version, and evaluate data at scale
• Collaborate with pre-training, post-training, and partnerships teams to drive improvements

Skills

LLM training

Data curation

ML pipelines

Cross-functional collab

Education

BS/MS/PhD in CS/ML or related

Job description

Reflection is a research lab focused on making intelligence open and accessible. The Data Flywheel team closes the gap between benchmark performance and real-world usefulness, building signals, data, and feedback loops to improve model behavior.

You will identify high-value data sources, onboard new data, and design evaluations that turn signals into measurable improvements across model generations. This is a hands-on, collaborative role in a fast-paced startup environment.