Staff ML Engineer: Data & Evaluation for Open Models

reflectionai; New York (NY); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals in office

Parental leave 22 weeks

Unlimited vacation in US / 30 days UK

Sponsorship visas

Team building events

Job summary

Reflection is seeking a hands-on data/ML role on the Data Flywheel team. You will turn ambiguous model behaviors into measurable improvements through evaluation design, data collection, and training interventions, collaborating with researchers, engineers, and external partners.

You will build pipelines to ingest and validate data at scale, design graders and feedback loops, and translate insights into concrete datasets and reward signals.

Qualifications

• Degree in Computer Science, ML or related field (or equivalent practical experience).
• Deep understanding of LLM training/evaluation with hands-on experience in evaluation design and data curation.
• Strong software engineering skills and experience building automated data/evaluation pipelines or large-scale ML systems.
• Proven track record of owning high-impact projects end to end in a fast-paced setting.

Responsibilities

• Identify high-value data sources and translate use cases into evaluations.
• Onboard new data sources, validate quality, and integrate into production pipelines.
• Design and build evaluations, graders, and feedback loops for real-world measurability.
• Analyze model performance and translate insights into datasets and training signals.
• Develop data strategies with vendors for capabilities gaps.
• Build scalable data ingestion, inspection, and evaluation pipelines.
• Collaborate with research, engineering, operations, and product teams to drive model progress.

Skills

LLM evaluation

ML pipelines

Project ownership

Cross-functional collaboration

Adaptability

Collaboration across teams

Education

BS/MS/PhD in CS/ML or related

Job description

Reflection is seeking a hands-on data/ML role on the Data Flywheel team. You will turn ambiguous model behaviors into measurable improvements through evaluation design, data collection, and training interventions, collaborating with researchers, engineers, and external partners.

You will build pipelines to ingest and validate data at scale, design graders and feedback loops, and translate insights into concrete datasets and reward signals.