Data Flywheel Engineer for Open Models & Evaluation

Reflection; New York (NY); Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Top-tier comp

Stock options

Health insurance

Meals provided

Parental leave

Unlimited PTO

Visa sponsorship

Team events

Job summary

Reflection is a research lab focused on open, accessible AI. This role sits at the intersection of research and deployment within the Data Flywheel team, taking ambiguous model behaviors from initial observation to measurable improvements across evaluation, data, and live deployments.

You'll collaborate with researchers, engineers, customers, and partners to turn signals into robust evaluation pipelines, data strategies, and targeted training interventions that drive real-world model progress.

Qualifications

• Degree (BS, MS, or PhD) in Computer Science, Machine Learning, or related discipline, or equivalent practical experience
• Hands-on experience in LLM training and evaluation, including evaluation design, data curation, reinforcement learning, or reward design
• Strong software engineering skills and experience building automated data/evaluation pipelines or large-scale ML systems

Responsibilities

• Identify high-value data sources and partnership opportunities for evaluations
• Bring new data sources online from partner conversations to production pipelines
• Design and build evaluations, graders, and feedback loops for real-world model behaviors
• Analyze model performance and failure modes; translate insights into datasets and training signals
• Develop human and synthetic data strategies for capabilities with insufficient existing data
• Build infrastructure and pipelines to ingest, inspect, version, and evaluate data at scale
• Collaborate with pre-training, post-training, applied, and partnership teams to turn signals into improvements

Skills

LLM training

Evaluation design

Data curation

RL concepts

Reward design

Data pipelines

Software engineering

Education

BS/MS/PhD in CS/ML or equivalent

Tools

Python

PyTorch

TensorFlow

Airflow

Job description

Reflection is a research lab focused on open, accessible AI. This role sits at the intersection of research and deployment within the Data Flywheel team, taking ambiguous model behaviors from initial observation to measurable improvements across evaluation, data, and live deployments.

You'll collaborate with researchers, engineers, customers, and partners to turn signals into robust evaluation pipelines, data strategies, and targeted training interventions that drive real-world model progress.