Data Quality Engineer, LLM Post-Training Pipelines

Sierra Ventures; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health insurance and wellness

Meals in office

Parental leave

Unlimited vacation

Visa sponsorship

Team events

Job summary

Reflection is a research lab focused on making intelligence open and accessible for everyone to use and build on. As part of the Data Team, you will ensure the data used to train and evaluate models meets high standards for quality, reliability, and downstream impact, shaping performance on critical capabilities.

You will collaborate with world-class researchers, design automated QA methods, and scale pipelines to support post-training data at scale, with emphasis on agentic use cases and robust

Qualifications

• Proficiency in Python and building ML/LLM workflows.
• Experience working with large datasets and automated evaluation or quality-checking systems.
• Familiarity with how LLMs work and can describe how models are trained and evaluated.
• Excellent communication skills with the ability to clearly articulate complex technical concepts across teams.
• Comfortable working autonomously, owning problems end-to-end, and collaborating effectively with researchers, engineers, and operations partners.

Responsibilities

• Own upstream data quality for LLM post-training and evaluation by analyzing expert-developed datasets and operationalizing quality standards for reasoning, alignment, and agentic use cases.
• Partner closely with research and post-training teams to translate requirements into measurable quality signals, and provide actionable feedback to external data vendors.
• Design, validate, and scale automated QA methods, including LLM-as-a-Judge frameworks, to reliably measure data quality across large campaigns.
• Build reusable QA pipelines that reliably deliver high-quality data to post-training teams for model training and evaluation.
• Monitor and report on data quality over time, driving continuous iteration on quality standards, processes, and acceptance criteria.

Skills

Python

ML/LLM workflows

Large datasets

Communication

Job description

Reflection is a research lab focused on making intelligence open and accessible for everyone to use and build on. As part of the Data Team, you will ensure the data used to train and evaluate models meets high standards for quality, reliability, and downstream impact, shaping performance on critical capabilities.

You will collaborate with world-class researchers, design automated QA methods, and scale pipelines to support post-training data at scale, with emphasis on agentic use cases and robust