Remote Factuality Failure Analysis Reviewer

AuraOne; United States; On-site; USD 34,440 - 55,104; Full time

Job summary

AuraOne seeks a remote, contractor-based AI data reviewer to evaluate factuality outputs, apply multi-page rubrics, and assign severity tags. You will compare responses, write clear rationales, and flag hallucinations, following established rubrics and calibrations against gold standards.

Ideal candidates have evaluation or annotation experience, attention to detail, and reliable async availability for at least 10 hours per week.

Qualifications

• Proven experience evaluating factuality and labeling according to rubrics.
• Ability to apply multi-page rubrics consistently across large batches.
• Clear written reasoning naming the issue and rubric clause.
• Strong attention to detail and error spotting in prompts.

Responsibilities

• Evaluate model outputs against rubric and assign severity tags.
• Compare paired responses and select stronger with rationale.
• Tag hallucinations and unsafe content with policy categories.
• Capture ambiguous prompts and route for rubric updates.
• Calibrate reviewer quality against gold-standard examples weekly.
• Document recurring failure modes for modeling team.

Skills

Factuality evaluation experience

Rubric consistency

Clear written reasoning

Attention to detail

Part-time availability

Job description

AuraOne seeks a remote, contractor-based AI data reviewer to evaluate factuality outputs, apply multi-page rubrics, and assign severity tags. You will compare responses, write clear rationales, and flag hallucinations, following established rubrics and calibrations against gold standards.

Ideal candidates have evaluation or annotation experience, attention to detail, and reliable async availability for at least 10 hours per week.