Remote AI Evaluation Rubric Reviewer | Labeling & Calibration

AI Trainer Jobs; United States; Remote; USD 28,000 - 62,000; Full time

Job summary

AI Trainer Jobs is seeking an AI Evaluation Rubric Writer for a remote, independent contractor role. You will review rubric writer prompts and responses, label edge cases, and provide structured feedback for retraining.

Responsibilities include evaluating outputs against a versioned rubric, annotating with severity tags, and calibrating against gold-standard samples weekly. Strong reasoning and attention to detail are essential; remote candidates from the US are eligible.

Qualifications

• Prior evaluation, annotation, or human-rater experience on ai evaluation rubric writer evaluation or adjacent content for AI Evaluation Rubric Writer work.

Responsibilities

• Evaluate ai evaluation rubric writer evaluation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and provide a written rationale for the stronger answer.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route to program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

AI Evaluation Rubric Writer evaluation

Cross-cutting evaluation

AI evaluation

Rubric writing

Expert review

Job description

AI Trainer Jobs is seeking an AI Evaluation Rubric Writer for a remote, independent contractor role. You will review rubric writer prompts and responses, label edge cases, and provide structured feedback for retraining.

Responsibilities include evaluating outputs against a versioned rubric, annotating with severity tags, and calibrating against gold-standard samples weekly. Strong reasoning and attention to detail are essential; remote candidates from the US are eligible.