Remote AI Data Reviewer: Document Annotation & Evaluation

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

AuraOne is seeking a remote CNC Manufacturing Documentation Reviewer to evaluate document annotation model outputs against a rubric, compare paired responses, and provide structured feedback for retraining. You will label edge cases, flag issues, and help calibrate reviewers weekly, enabling improvements to the modeling system.

Role is a contractor position with remote US eligibility. Responsibilities include maintaining quality scores, applying rubrics consistently across batches, and

Qualifications

• Prior evaluation, annotation, or human-rater experience on document annotation.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate document annotation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route to the program team for rubric updates.
• Calibrate reviewer-quality scores against gold-standard examples weekly.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Document annotation

Education

Background in linguistics, content moderation, or trust & safety review

Job description

AuraOne is seeking a remote CNC Manufacturing Documentation Reviewer to evaluate document annotation model outputs against a rubric, compare paired responses, and provide structured feedback for retraining. You will label edge cases, flag issues, and help calibrate reviewers weekly, enabling improvements to the modeling system.

Role is a contractor position with remote US eligibility. Responsibilities include maintaining quality scores, applying rubrics consistently across batches, and