Historical Reasoning AI Evaluator — Remote Quality Feedback

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

AI Trainer Jobs seeks a Historical Reasoning AI Evaluator for AuraOne's quality rubric. This remote contractor role involves reviewing prompts and responses, comparing paired outputs, labeling edge cases, and providing structured feedback to retrain the model.

Responsibilities include evaluating outputs against the rubric, selecting the stronger answer with rationale, tagging issues, and maintaining calibration against gold-standard examples each week.

Qualifications

• Prior evaluation, annotation, or human-rater experience on historical reasoning AI evaluation or adjacent content.

Responsibilities

• Evaluate historical reasoning AI evaluation model outputs against a versioned rubric and assign severity tags for Historical Reasoning AI Evaluator assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.

Skills

Evaluation experience

Annotation experience

Written reasoning

Attention to detail

Async availability

Job description

AI Trainer Jobs seeks a Historical Reasoning AI Evaluator for AuraOne's quality rubric. This remote contractor role involves reviewing prompts and responses, comparing paired outputs, labeling edge cases, and providing structured feedback to retrain the model.

Responsibilities include evaluating outputs against the rubric, selecting the stronger answer with rationale, tagging issues, and maintaining calibration against gold-standard examples each week.