LLM Data Quality Engineer — Pre-Training

reflectionai; San Francisco, New York (CA, NY); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals provided in-office

Parental leave

Unlimited vacation

Job summary

Reflection AI seeks a role in Data QA for LLM pre-training, focusing on improving the quality and reliability of data used to train models. You will shape data standards, partner with researchers to define signals, and scale automated QA methods across large data campaigns.

You will build reusable pipelines, collaborate with researchers and engineers, and monitor quality metrics to drive continual improvements in data curation and model performance.

Qualifications

• Proficiency in Python and building ML/LLM workflows.
• Experience with large datasets and automated evaluation systems.
• Understanding of how LLMs are trained and evaluated.

Responsibilities

• Own upstream data quality for LLM pre-training and build QA pipelines.
• Translate requirements into measurable quality signals and provide feedback to data vendors.
• Design, validate, and scale automated QA methods for large data campaigns.
• Deliver high-quality data to pre-training teams for model training.
• Monitor data quality over time and drive continual improvement.

Skills

Python

ML workflows

LLM understanding

Debugging

Communication

Tools

scalable code

Job description

Reflection AI seeks a role in Data QA for LLM pre-training, focusing on improving the quality and reliability of data used to train models. You will shape data standards, partner with researchers to define signals, and scale automated QA methods across large data campaigns.

You will build reusable pipelines, collaborate with researchers and engineers, and monitor quality metrics to drive continual improvements in data curation and model performance.