AI Output Evaluator - Consumer Data Quality (Remote)

AuraOne; United States; Remote; USD 28,000 - 55,000; Part time

Job summary

AuraOne is seeking a Remote — US-eligible Consumer Reviewer for AI Output Evaluation as a CONTRACTOR. You will evaluate model outputs against a rubric, label edge cases, and write structured feedback to help retrain the model. The role emphasizes explaining reasoning and maintaining calibration across batches.

Responsibilities include comparing responses, tagging unsafe content, and documenting recurring failure modes for rubric improvements. An async, part-time commitment is needed.

Qualifications

• Experience evaluating consumer evaluation or adjacent content using rubrics.
• Clear written reasoning that names the issue and the rubric clause applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate consumer evaluation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibration against gold-standard examples.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Consumer evaluation

Job description

AuraOne is seeking a Remote — US-eligible Consumer Reviewer for AI Output Evaluation as a CONTRACTOR. You will evaluate model outputs against a rubric, label edge cases, and write structured feedback to help retrain the model. The role emphasizes explaining reasoning and maintaining calibration across batches.

Responsibilities include comparing responses, tagging unsafe content, and documenting recurring failure modes for rubric improvements. An async, part-time commitment is needed.