Data Quality Engineer — LLM Post-Training & QA Pipelines

Reflection AI Ltd; New York (NY); On-site; USD 140,000 - 190,000; Full time

Benefits offered by this job

Top-tier compensation

Stock options

Health & wellness

Meals in office

Life & family benefits

Unlimited PTO in US / 30 days in UK

Sponsorship support

Team building

Job summary

Reflection AI Ltd. is seeking a data-focused engineer to own upstream data quality for LLM post-training and evaluation, shaping how models perform on critical capabilities.

You will collaborate with researchers and post-training teams to translate requirements into measurable quality signals and build scalable QA pipelines for large data campaigns. You will design automated quality checks and contribute to continuous improvement of data standards, with hands-on work in Python, ML workflows, and

Qualifications

• Strong engineering fundamentals with experience building data pipelines, QA systems, or evaluation workflows for post-training data and agentic environments.
• Detail-oriented with an analytical mindset, able to identify failure modes, inconsistencies, and subtle issues that affect data quality.
• Solid understanding of how data quality impacts training (SFT and RL) and evaluation, with the ability to translate quality concerns into concrete signals, decisions, and feedback.
• Experience designing and validating automated quality checks, including rule-based systems, statistical methods, or model-assisted approaches such as LLM-as-a-Judge.
• Comfortable working autonomously, owning problems end-to-end, and collaborating effectively with researchers, engineers, and operations partners

Responsibilities

• Own upstream data quality for LLM post-training and evaluation by analyzing expert-developed datasets and operationalizing quality standards for reasoning, alignment, and agentic use cases.
• Partner closely with research and post-training teams to translate requirements into measurable quality signals, and provide actionable feedback to external data vendors.
• Design, validate, and scale automated QA methods, including LLM-as-a-Judge frameworks, to reliably measure data quality across large campaigns.
• Build reusable QA pipelines that reliably deliver high-quality data to post-training teams for model training and evaluation.
• Monitor and report on data quality over time, driving continuous iteration on quality standards, processes, and acceptance criteria

Skills

Python

ML workflows

Data pipelines

QA systems

Evaluation workflows

Job description

Reflection AI Ltd. is seeking a data-focused engineer to own upstream data quality for LLM post-training and evaluation, shaping how models perform on critical capabilities.

You will collaborate with researchers and post-training teams to translate requirements into measurable quality signals and build scalable QA pipelines for large data campaigns. You will design automated quality checks and contribute to continuous improvement of data standards, with hands-on work in Python, ML workflows, and