Remote AI Data Reviewer: Windows Premiere (Generalist)

AuraOne, Inc.; United States; Remote; USD 41,000 - 55,000; Full time

Benefits offered by this job

Remote work

Job summary

AuraOne, Inc. is seeking an experienced remote evaluation generalist to review application users premiere on Windows outputs and annotate with rubric-based labels. You will compare responses, identify edge cases, and document issues for retraining, working as a contractor with flexible weekly hours.

Role focuses on ensuring quality and consistency across long batches, with emphasis on clear reasoning and precise tagging of failures. US-based remote work is supported.

Qualifications

• Experience evaluating model outputs using rubric-based guidelines.
• Ability to apply rubrics consistently across long batches.
• Clear written reasoning citing rubric clauses.
• Strong attention to detail and ability to flag prompt issues.
• Experience with inter-rater calibration is a plus.

Responsibilities

• Evaluate model outputs against the versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibration against gold-standard examples.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Generalist evaluation experience

Job description

AuraOne, Inc. is seeking an experienced remote evaluation generalist to review application users premiere on Windows outputs and annotate with rubric-based labels. You will compare responses, identify edge cases, and document issues for retraining, working as a contractor with flexible weekly hours.

Role focuses on ensuring quality and consistency across long batches, with emphasis on clear reasoning and precise tagging of failures. US-based remote work is supported.