Remote AI Agent QA & Workflow Evaluator

AuraOne; United States; À distance; USD 70 000 - 135 000; Plein temps

Résumé du poste

AuraOne is seeking an Everyday AI Agent Tester specialist operations contractor for remote US-eligible work. You will review AI outputs against workflows, policy, and playbooks, grade tone and escalation, and document next steps for model training.

Ideal candidates have hands-on experience with real teams performing everyday AI agent tester operations, can apply multi-page rubrics consistently, and communicate clear reasoning about policy or workflow.

Qualifications

• Experience in everyday AI agent tester operations on real teams.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning that names the policy or workflow being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsabilités

• Review AI outputs against current everyday AI agent tester specialist operations workflows, playbooks, and firm policy for Everyday AI Agent Tester assignments.
• Grade tone, escalation logic, and stakeholder fit on a structured rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputed treatments against published playbooks or firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Connaissances

Operational review

Policy adherence

Workflow judgment

Stakeholder communication

Everyday AI Agent Tester specialist

AI model evaluation

Description du poste

AuraOne is seeking an Everyday AI Agent Tester specialist operations contractor for remote US-eligible work. You will review AI outputs against workflows, policy, and playbooks, grade tone and escalation, and document next steps for model training.

Ideal candidates have hands-on experience with real teams performing everyday AI agent tester operations, can apply multi-page rubrics consistently, and communicate clear reasoning about policy or workflow.