Remote Frontier AI/Software Evaluator & Feedback Specialist

AI Trainer Jobs; United States; Remote; USD 110,000 - 165,000; Part time

Job summary

AI Trainer Jobs is seeking a remote Software / AI / IT / data Evaluator to review prompts and responses against a quality rubric. Reviewers compare outputs, label edge cases, and provide structured feedback for model retraining.

The role involves evaluating model outputs, tagging issues, and calibrating against gold standards on a weekly cadence. Flexible, contractor-based, remote from the US with independent work pacing.

Qualifications

• Prior evaluation, annotation, or human-rater experience on software/ai/data evaluation or adjacent content for Software/AI/IT/data Evaluator work.

Responsibilities

• Evaluate model outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger answer with written rationale.
• Label hallucinations and unsafe content with structured tags.
• Capture ambiguous prompts and route to program team for rubric updates.
• Maintain reviewer-quality scores by calibration against gold-standard examples weekly.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Software / AI / IT / data evaluation

Job description

AI Trainer Jobs is seeking a remote Software / AI / IT / data Evaluator to review prompts and responses against a quality rubric. Reviewers compare outputs, label edge cases, and provide structured feedback for model retraining.

The role involves evaluating model outputs, tagging issues, and calibrating against gold standards on a weekly cadence. Flexible, contractor-based, remote from the US with independent work pacing.