Remote Annotation Guideline Evaluator & Rubric Specialist

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AI Trainer Jobs is seeking an Annotation Guideline Data Specialist for a remote evaluation track. You will review annotation guideline data evaluation prompts and responses against AuraOne's quality rubric, compare outputs, label edge cases, and provide structured feedback for retraining.

The role is remote, US-eligible, contractor, hourly pay to be confirmed after the interview. You will help turn evaluation outputs into auditable labels, rationales, and regression cases for the modeling team,

Qualifications

• Prior evaluation, annotation, or human-rater experience on annotation guideline data evaluation.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.
• Experience with inter-rater agreement metrics and calibration cycles.
• Background in linguistics, content moderation, or trust & safety review.

Responsibilities

• Evaluate annotation guideline data evaluation model outputs against a versioned rubric and assign severity tags for Annotation Guideline Data Specialist assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.

Skills

Annotation evaluation

Written reasoning

Attention to detail

Availability

Rubric application

Linguistics background

Inter-rater metrics

Job description

AI Trainer Jobs is seeking an Annotation Guideline Data Specialist for a remote evaluation track. You will review annotation guideline data evaluation prompts and responses against AuraOne's quality rubric, compare outputs, label edge cases, and provide structured feedback for retraining.

The role is remote, US-eligible, contractor, hourly pay to be confirmed after the interview. You will help turn evaluation outputs into auditable labels, rationales, and regression cases for the modeling team,