Research Engineer, QC Automation

Dreams 2 Reality Recruitment; San Francisco (CA); On-site; USD 100,000 - 200,000; Full time

Job summary

Dreams 2 Reality Recruitment is seeking a Research Engineer, QC Automation in the San Francisco Bay Area to build robust QC infrastructure for AI training data. You will design automated QA/QC pipelines, define quality standards, and collaborate with data vendors to identify and fix agent failure modes.

The role emphasizes autonomy, rapid learning, and turning ambiguous quality requirements into scalable, measurable systems within an early-stage startup environment.

Qualifications

• Strong proficiency in Python, Docker, and Linux.
• Experience building scalable data-validation pipelines and automated QA/QC systems.
• Ability to design benchmarks and evaluations for AI data quality.
• Comfort working in an early-stage startup environment and taking ownership.
• Strong written and verbal communication; curious and fast learner.

Responsibilities

• Build quality-control systems grounded in human judgment and data quality.
• Define, formalize, and enforce quality standards for training data.
• Design experiments, benchmarks, and metrics to evaluate agent performance.
• Partner with data vendors to identify failure modes and improve data-generation processes.
• Build systems for auditing supplier datasets, including sampling and validation approaches.
• Integrate QC insights into HUD infrastructure and vendor portal to reduce anomalies.

Skills

Python

Data quality

QA/QC

Experiment design

Statistics

Learning agility

Startup experience

Communication

Tools

Python

Docker

Linux

Job description

Location: San Francisco Bay Area, CA — On-site
Employment: Full-time
Compensation: $100,000–$200,000
Visa: Relocation and visa support available for strong candidates
Experience: Technical aptitude and learning potential matter more thanyears of experience.

About the Role

HUD builds infrastructure for companies creating trainingdata for AI agents. As demand grows, we need robust systems that can maintainand scale data quality.

As a Research Engineer, QC Automation, you’ll buildthe systems that make that possible. You’ll develop automated quality-controlinfrastructure grounded in human judgment and a deep understanding of whatmakes training data useful—not simply by relying on LLMs to judge other LLMs.

This is a highly autonomous role for someone who enjoysambiguous problems, learns quickly, and can turn unclear quality requirementsinto measurable, scalable systems.

What You’ll Do

• Build quality-controlsystems grounded in human judgment and a deep understanding of dataquality, with limited reliance on LLM-based evaluation.
• Define,formalize, and enforce quality standards for training data.
• Designexperiments, benchmarks, and metrics to evaluate agent performance.
• Partnerwith data vendors to identify agent failure modes, debug qualityissues, and improve data-generation processes.
• Buildsystems for auditing supplier datasets, including samplingstrategies, rule-based validation, model-assisted validation, and feedbackloops.
• IntegrateQC insights into HUD’s infrastructure and data-vendor portal to reduce anomalies,inconsistencies, and edge cases.
• Workacross unfamiliar domains and turn loosely defined quality problems into reliable,repeatable processes.

What We’re Looking For

• Strongproficiency in Python, Docker, and Linux environments.
• Excellentjudgment about what constitutes good data and how to measure it.
• Genuinecuriosity about unfamiliar domains, with the ability to ask the rightquestions to develop a deep understanding quickly.
• Experiencebuilding scalable data-validation pipelines, automated QA/QC systems,or similar infrastructure without a prescribed roadmap.
• Experiencedesigning or working with benchmarks and evaluations.
• Comfortworking in an early-stage startup environment, taking ownership andexecuting independently.
• Atrack record of learning quickly and tackling problems that don't comewith clear instructions.

Strong Signals

• Knowledgeof statistics and experimental design.
• Strongwritten and verbal communication skills.
• Comfortdesigning metrics, experiments, and QA/QC processes.
• Abilityto construct tasks for new evaluations and benchmarks.
• Comfortoperating in unstructured problem spaces.
• Atendency to dig into the underlying problem rather than reaching for thefirst available tool or solution.