Remote Model Failure Analyst - AI Output Reviewer

AuraOne; United States; On-site; USD 48,000 - 90,000; Full time

Job summary

AuraOne is seeking a remote Model Failure Analyst to evaluate AI outputs across specialist operations workflows, grading for workflow correctness, policy adherence, and stakeholder fit. You will document the right next steps to inform the modeling team and help improve training data and guidelines.

The role emphasizes rigorous rubric-based assessment, escalation decisions, and careful attention to operational risk, with a contractor-like pace and US-only eligibility for remote work.

Qualifications

• Direct working experience in model failure analyst specialist operations on real teams for Model Failure Analyst work.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the policy or workflow being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Review AI outputs against current model failure analyst specialist operations workflows, playbooks, and firm policy for Model Failure Analyst assignments.
• Grade tone, escalation logic, and stakeholder fit on a structured rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputed treatments against published playbooks or firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Job description

AuraOne is seeking a remote Model Failure Analyst to evaluate AI outputs across specialist operations workflows, grading for workflow correctness, policy adherence, and stakeholder fit. You will document the right next steps to inform the modeling team and help improve training data and guidelines.

The role emphasizes rigorous rubric-based assessment, escalation decisions, and careful attention to operational risk, with a contractor-like pace and US-only eligibility for remote work.