LLM Data Quality Engineer – Pre-Training

Reflection AI Ltd; New York (NY); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness benefits

Meals in office

Parental leave (22 weeks)

Job summary

Reflection AI Ltd. is seeking a data-focused engineer to own upstream data quality for LLM pre-training. You will collaborate with world-class researchers to translate data requirements into measurable signals and scale automated QA across large campaigns.

You will design reusable QA pipelines, monitor data quality over time, and work with researchers, engineers, and operations partners to drive continuous improvement in data standards and acceptance criteria.

Qualifications

• Proficiency in Python and building ML/LLM workflows.
• Experience with large datasets and automated evaluation or quality-checking systems.
• Familiarity with how LLMs work and can describe how models are trained and evaluated.
• Excellent communication skills across teams.

Responsibilities

• Own upstream data quality for LLM pre-training; as a specialist or generalist across languages and modalities.
• Partner with research and pre-training teams to translate requirements into measurable quality signals and provide feedback to data vendors.
• Design, validate, and scale automated QA methods to measure data quality across large campaigns.
• Build reusable QA pipelines that deliver high-quality data to pre-training teams.
• Monitor and report on data quality over time, driving continuous iteration on quality standards.

Skills

Python

LLM workflows

Large datasets

Communication

Job description

Reflection AI Ltd. is seeking a data-focused engineer to own upstream data quality for LLM pre-training. You will collaborate with world-class researchers to translate data requirements into measurable signals and scale automated QA across large campaigns.

You will design reusable QA pipelines, monitor data quality over time, and work with researchers, engineers, and operations partners to drive continuous improvement in data standards and acceptance criteria.