Remote Frontier Model Evaluator - Failure Analysis Reviewer

AI Trainer Jobs; United States; Remote; USD 55,000 - 96,000; Part time

Job summary

AuraOne seeks an Instruction Following Failure Analysis Reviewer to remotely evaluate prompts and responses against its quality rubric. You will compare paired outputs, label edge cases, and write structured feedback to retrain the model.

Responsibilities include evaluating outputs, selecting the stronger answer with rationale, tagging issues, and routing ambiguous prompts back for rubric updates. This role is a contractor and US-remote eligible.

Qualifications

• Experience evaluating instruction following failure analyses against rubrics.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning naming issues and rubric clauses.
• Strong attention to detail and ability to flag a prompt problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate model outputs against the rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route to program team for rubric updates.

Skills

Evaluation experience

Rubrics familiarity

Clear written reasoning

Attention to detail

Async availability

Job description

AuraOne seeks an Instruction Following Failure Analysis Reviewer to remotely evaluate prompts and responses against its quality rubric. You will compare paired outputs, label edge cases, and write structured feedback to retrain the model.

Responsibilities include evaluating outputs, selecting the stronger answer with rationale, tagging issues, and routing ambiguous prompts back for rubric updates. This role is a contractor and US-remote eligible.