Remote AI Data Reviewer — Content Evaluation (Contract)

AuraOne, Inc.; United States; Remote; USD 41,000 - 55,000; Full time

Job summary

AuraOne, Inc. seeks a remote Evaluation Generalist to review Blender on Windows outputs, apply a multi-page rubric, and provide structured feedback for retraining. The role emphasizes precise labeling of issues, comparisons of model responses, and maintaining calibration against gold standards.

Applicants should have experience in evaluation or annotation of similar content, with reliable weekly availability and strong attention to detail. This is a remote US-eligible contractor position.

Qualifications

• Prior evaluation, annotation, or human-rater experience on Application Users - Blender on Windows - Generalist work.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and rubric clause being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate model outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples weekly.
• Document recurring failure modes for future training.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Evaluation experience

Job description

AuraOne, Inc. seeks a remote Evaluation Generalist to review Blender on Windows outputs, apply a multi-page rubric, and provide structured feedback for retraining. The role emphasizes precise labeling of issues, comparisons of model responses, and maintaining calibration against gold standards.

Applicants should have experience in evaluation or annotation of similar content, with reliable weekly availability and strong attention to detail. This is a remote US-eligible contractor position.