Data Flywheel Engineer: Real-World ML Data & Evaluation

Sierra Ventures; San Francisco (CA); On-site; USD 180,000 - 280,000; Full time

Benefits offered by this job

Stock options

Health & wellness

Meals provided in office

Job summary

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. The Data Flywheel team builds signals, data, and feedback loops to turn model usage into measurable improvements.

This hands-on role sits at the intersection of research and deployment, collaborating with researchers, engineers, customers, partners, vendors, and the open-source community to drive real-world impact.

Qualifications

• Degree (BS, MS, or PhD) in Computer Science, Machine Learning, or related discipline, or equivalent practical experience
• Deep technical understanding of LLM training and evaluation, with hands-on experience in evaluation design, data curation, reinforcement learning, or reward design
• Strong software engineering skills and experience building automated data/evaluation pipelines or large-scale ML systems
• A track record of owning high-impact projects end to end, navigating ambiguity, and adapting quickly as priorities change
• A highly collaborative, action-oriented approach and excitement about defining how a new frontier lab measures and accelerates model progress
• High agency and thrive in a fast-paced startup environment; bias for impact over process
• Enjoy collaborating across research, engineering, operations, and product disciplines

Responsibilities

• Identify high-value data sources and partnership opportunities, deeply understand the underlying use cases, and translate them into representative evaluations
• Bring new data sources online, from initial partner conversations and data scoping through quality validation and integration into production evaluation and training pipelines, where appropriate
• Design and build evaluations, graders, and feedback loops that make priority real-world model behaviors measurable
• Analyze model performance and failure modes, then translate those insights into targeted datasets, reward signals, and training interventions
• Develop human and synthetic data strategies for capabilities where existing data is insufficient, including designing and running evaluation and data-collection programs with vendors
• Build the infrastructure and pipelines needed to ingest, inspect, version, and evaluate data reliably at scale
• Collaborate closely with pre-training, post-training, applied, and partnership teams to turn new signals into measurable model improvements

Skills

LLM evaluation

Data pipelines

Model evaluation

Collaborative mindset

Ambiguity navigation

Education

BS/MS/PhD in CS or related

Tools

Python

Evaluation tooling

Job description

Reflection is a research lab making intelligence open and accessible for everyone to use, customize, and build on. The Data Flywheel team builds signals, data, and feedback loops to turn model usage into measurable improvements.

This hands-on role sits at the intersection of research and deployment, collaborating with researchers, engineers, customers, partners, vendors, and the open-source community to drive real-world impact.