Remote Fact-Check Workflow Evaluator (Rubric-Based)

AI Trainer Jobs; United States; Remote; USD 28,000 - 55,000; Part time

Job summary

AI Trainer Jobs is seeking a remote Fact-Check Workflow Task Evaluator to review prompts and responses against AuraOne's rubric. This contractor role involves comparing outputs, labeling edge cases, and providing structured feedback for retraining.

You will evaluate model outputs, tag issues, calibrate with gold standards, and maintain quality scores. Flexible hourly work with US eligibility and remote setup.

Qualifications

• Prior evaluation, annotation, or human-rater experience on fact check workflow task evaluation or adjacent content.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Strong attention to detail and ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.
• Comfort applying multi-page rubrics consistently across long batches.
• Experience with inter-rater agreement metrics and calibration cycles.

Responsibilities

• Evaluate outputs against rubric and assign severity tags for Fact-Check Workflow Task Evaluator assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.

Skills

Evaluation experience

Clear reasoning

Attention to detail

Async availability

Inter-rater calibration

Multi-page rubrics

Tools

Web research

Source grounding

Browser automation

Job description

AI Trainer Jobs is seeking a remote Fact-Check Workflow Task Evaluator to review prompts and responses against AuraOne's rubric. This contractor role involves comparing outputs, labeling edge cases, and providing structured feedback for retraining.

You will evaluate model outputs, tag issues, calibrate with gold standards, and maintain quality scores. Flexible hourly work with US eligibility and remote setup.