Frontier Model Evaluation Reviewer: Generalization Failures

AI Trainer Jobs; United States; Remote; USD 34,000 - 48,000; Part time

Job summary

AuraOne is seeking a Capability Generalization Failure Analysis Reviewer for a remote contractor role. You will evaluate frontier model outputs, compare paired responses, label edge cases, and provide structured feedback to retrain the system.

Responsibilities include scoring against a versioned rubric, tagging issues, and documenting rationale for program reviews. The role requires strong attention to detail and reliable async availability for 10+ hours weekly.

Qualifications

• Prior evaluation, annotation, or human-rater experience in capability generalization failure analysis or related content.
• Ability to apply rubrics consistently across long batches.
• Clear written reasoning that names the issue and rubric clause.

Responsibilities

• Evaluate model outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger with a rationale.
• Label hallucinations, instruction-following, and unsafe content with tags.
• Capture ambiguous prompts and route to rubric updates.

Skills

Background in linguistics, content/mod

Inter-rater agreement metrics

Subject-matter error spotting

Job description

AuraOne is seeking a Capability Generalization Failure Analysis Reviewer for a remote contractor role. You will evaluate frontier model outputs, compare paired responses, label edge cases, and provide structured feedback to retrain the system.

Responsibilities include scoring against a versioned rubric, tagging issues, and documenting rationale for program reviews. The role requires strong attention to detail and reliable async availability for 10+ hours weekly.