Staff ML Evaluation & Data Engineer

Reflection; New York (NY); On-site; USD 190,000 - 270,000; Full time

Benefits offered by this job

Stock options

Health & wellness benefits

Daily meals in office

Parental leave

Unlimited vacation in US

Visa sponsorship

Job summary

Reflection is seeking a hands-on data-centric role at the intersection of research and deployment. You will own end-to-end evaluation workflows for open models, turning signals into measurable improvements across evaluation, data collection, and live deployments.

Work closely with researchers, engineers, and partners to scope data, build pipelines, and drive impactful improvements in future model generations. A collaborative, fast-paced startup environment awaits.

Qualifications

• Degree in Computer Science, Machine Learning, or related discipline, or equivalent practical experience.
• Hands-on experience in LLM training and evaluation, including evaluation design and data curation.
• Strong software engineering skills and experience building automated data/evaluation pipelines or large-scale ML systems.

Responsibilities

• Identify high-value data sources and translate use cases into evaluations.
• Bring new data sources online from partnerships through production integration.
• Design evaluations, graders, and feedback loops to measure real-world model behaviors.
• Analyze model performance and failure modes to drive targeted datasets and training signals.
• Develop human and synthetic data strategies for data-scarce capabilities.
• Build scalable data ingest, versioning, and evaluation pipelines.
• Collaborate with pre-training, post-training, and partnership teams to drive model improvements.

Skills

LLM evaluation

Data pipelines

Software engineering

Collaboration

Education

BS/MS/PhD in CS/ML or equivalent

Tools

Python

ML frameworks (PyTorch/TensorFlow)

Job description

Reflection is seeking a hands-on data-centric role at the intersection of research and deployment. You will own end-to-end evaluation workflows for open models, turning signals into measurable improvements across evaluation, data collection, and live deployments.

Work closely with researchers, engineers, and partners to scope data, build pipelines, and drive impactful improvements in future model generations. A collaborative, fast-paced startup environment awaits.