Staff Data Flywheel Engineer — Open Models

Reflection AI; New York, San Francisco (NY, CA); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided

Parental leave

Unlimited PTO

Visa sponsorship

Team events

Job summary

Reflection AI is seeking a hands-on ML/data role at the intersection of research and deployment. You will translate model behaviors into measurable evaluations, design data-centric pipelines, and collaborate with researchers, engineers, and partners to drive real-world improvements.

You will own end-to-end data strategies, integrate diverse data sources, and build scalable evaluation systems that guide model development and deployment across teams.

Qualifications

• BS/MS/PhD in CS/ML or equivalent experience.
• Deep understanding of LLM training and evaluation.
• Hands-on experience in evaluation design, data curation, reinforcement learning, or reward design.
• Strong software engineering; experience building automated data/evaluation pipelines for ML systems.
• Proven track record delivering high-impact projects end to end.
• Collaborative and able to work across teams.
• Bias for impact in a fast-paced startup environment.

Responsibilities

• Identify high-value data sources and map use cases to evaluations.
• Onboard new data sources from partner talks to production pipelines.
• Design evaluations, graders, and feedback loops for model behaviors.
• Analyze performance and failure modes to inform datasets and rewards.
• Develop human and synthetic data strategies with vendors.
• Build scalable data ingestion, inspection, and evaluation pipelines.
• Collaborate with pre-training, post-training, applied, and partnership teams.

Skills

LLM evaluation

Data curation

Reinforcement learning

Reward design

Software engineering

Cross-functional collaboration

Education

BS/MS/PhD in CS/ML or equivalent

Tools

Python

PyTorch

TensorFlow

SQL

Job description

Reflection AI is seeking a hands-on ML/data role at the intersection of research and deployment. You will translate model behaviors into measurable evaluations, design data-centric pipelines, and collaborate with researchers, engineers, and partners to drive real-world improvements.

You will own end-to-end data strategies, integrate diverse data sources, and build scalable evaluation systems that guide model development and deployment across teams.