Research Engineer, QC Automation

Invictus Direct; San Francisco (CA); On-site; USD 100,000 - 200,000; Full time

Benefits offered by this job

Relocation assistance

Visa support

Job summary

Invictus Direct is seeking a Research Engineer, QC Automation to build scalable quality-control systems for reinforcement-learning training data and evaluations. You will develop pipelines that judge data quality, design experiments, and set standards while operating in a startup environment that values curiosity about new domains.

The role requires proficiency in Python, Docker, and Linux, plus strong data-validation experience and the ability to execute independently.

Qualifications

• Proficiency in Python, Docker, and Linux environments.
• Strong judgment about what good data means and how to measure it.
• Genuine curiosity about unfamiliar domains and skill at asking the questions needed to understand them.
• Experience building scalable data-validation pipelines or automated QA/QC systems without a prescribed roadmap.
• Experience with benchmarks and evaluations.
• Early-stage startup experience and independent execution.

Responsibilities

• Create QC systems grounded in true understanding and human judgment without relying heavily on LLMs
• Define and enforce quality standards for training data
• Design experiments and metrics to grade agent outputs
• Partner with data vendors to diagnose agent failure modes, debug quality issues, and improve data-generation processes
• Build systems for auditing supplier datasets, including sampling strategies, rule-based and model-assisted validation pipelines, and feedback loops
• Integrate QC learnings into infrastructure tools and the data-vendor portal to reduce anomalies, inconsistencies, and edge cases

Skills

Python

Docker

Linux

Data quality judgment

QA/QC systems

Experiment design

Independent execution

Startup experience

Tools

Data validation pipelines

Model evaluation

Job description

Location: San Francisco Bay Area, CA, US (On-site)
Schedule: Full-time
Salary: $100K–$200K

Company:
Our client is a Y Combinator-backed company building infrastructure to create reinforcement-learning training data and evaluations for frontier AI agents, along with a marketplace connecting that work to frontier labs. Its platform is used by frontier labs, Fortune 500 companies, and startups.

Description:
Our client is seeking a Research Engineer, QC Automation to automate quality control for training data created by companies using its infrastructure. This role will build systems that scale quality as the company meets continued strong demand, combining technical execution with strong judgment about data quality and genuine curiosity about unfamiliar domains.

Key Responsibilities:

• Create QC systems grounded in true understanding and human judgment without relying heavily on LLMs
• Define and enforce quality standards for training data
• Design experiments and metrics to grade agent outputs
• Partner with data vendors to diagnose agent failure modes, debug quality issues, and improve data-generation processes
• Build systems for auditing supplier datasets, including sampling strategies, rule-based and model-assisted validation pipelines, and feedback loops
• Integrate QC learnings into infrastructure tools and the data-vendor portal to reduce anomalies, inconsistencies, and edge cases

Qualifications:

Required:

• Proficiency in Python, Docker, and Linux environments
• Strong judgment about what good data means and how to measure it
• Genuine curiosity about unfamiliar domains and skill at asking the questions needed to understand them
• Experience building scalable data-validation pipelines or automated QA/QC systems without a prescribed roadmap
• Experience with benchmarks and evaluations
• Early-stage startup experience and independent execution

Technical aptitude and learning potential matter more than years of experience.

Preferred:

The following are considered strong signals:

• Knowledge of statistics
• Strong written and verbal communication
• Comfort designing metrics, experiments, and QA/QC processes
• Ability to construct tasks in new evaluations
• Comfort in unstructured problem spaces

Why Join Them?

• Build quality-control systems for reinforcement-learning training data and evaluations for frontier AI agents
• Help scale data quality as the company meets continued strong demand
• Work on infrastructure used by frontier labs, Fortune 500 companies, and startups
• Relocation and visa support are available for strong candidates