Remote AI Agent Tester & Workflow Evaluation Specialist

AuraOne Human Data

United States

Remote

USD 34,000 - 55,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

AuraOne, Inc. is seeking an AI Agent Tester specialist to evaluate outputs across ai agent tester specialist operations workflows.

This remote reviewer role involves grading workflow correctness, policy adherence, and stakeholder fit, and documenting the right next steps so the modeling team can train on it. You will review model outputs, apply a structured rubric, flag operational risk, and adjudicate disputes.

Qualifications

  • Direct working experience in ai agent tester specialist operations on real teams for AI Agent Tester, Business Workflow Evaluation work.
  • Comfort applying multi-page rubrics consistently across long batches.
  • Clear written reasoning that names the policy or workflow being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Responsibilities

  • Review AI outputs against current ai agent tester specialist operations workflows, playbooks, and firm policy for AI Agent Tester, Business Workflow Evaluation assignments.
  • Grade tone, escalation logic, and stakeholder fit on a structured rubric.
  • Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
  • Capture the right next step so the modeling team can train on it.
  • Adjudicate disputed treatments against published playbooks or firm guidance.
  • Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Direct working experience
Rubric application
Written reasoning
Attention to detail
Async availability

Job description

AuraOne, Inc. is seeking an AI Agent Tester specialist to evaluate outputs across ai agent tester specialist operations workflows.

This remote reviewer role involves grading workflow correctness, policy adherence, and stakeholder fit, and documenting the right next steps so the modeling team can train on it. You will review model outputs, apply a structured rubric, flag operational risk, and adjudicate disputes.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Agent QA & Workflow Evaluator
Remote AI Agent QA & Workflow Evaluator

Auraone • United States

Remote
USD 70,000 - 135,000
AI Agent QA Reviewer & Workflow Auditor (Remote)
AI Agent QA Reviewer & Workflow Auditor (Remote)

AuraOne, Inc. • United States

Remote
USD 28,000 - 55,000
Training opportunities
AI tooling familiarity
Cross-region collaboration
AI Agent Tester, Business Workflow Evaluation
AI Agent Tester, Business Workflow Evaluation

AuraOne Human Data • United States

Remote
USD 34,000 - 55,000
AI Agent Evaluator — Remote Rubric & Feedback Specialist
AI Agent Evaluator — Remote Rubric & Feedback Specialist

Auraone • United States

Remote
USD 34,000 - 55,000
Remote Retail AI Workflow Reviewer & Ops QA
Remote Retail AI Workflow Reviewer & Ops QA

AuraOne Human Data • United States

Remote
USD 28,000 - 55,000
Remote QA Specialist for AI Product Testing & Support
Remote QA Specialist for AI Product Testing & Support

AuraOne Human Data • United States

Remote
USD 28,000 - 48,000
Remote AI Evaluation Reviewer — Computer-Use Agent
Remote AI Evaluation Reviewer — Computer-Use Agent

AuraOne Human Data • United States

Remote
USD 34,000 - 62,000
AI Interaction Quality Reviewer & Rubric Specialist
AI Interaction Quality Reviewer & Rubric Specialist

AuraOne Human Data • United States

Remote
USD 28,000 - 55,000
Remote work
Remote AI Evaluation Specialist — Military Readiness
Remote AI Evaluation Specialist — Military Readiness

AuraOne Human Data • United States

Remote
USD 55,000 - 96,000
Remote AI QA Specialist: Coding Agent Evaluations
Remote AI QA Specialist: Coding Agent Evaluations

AuraOne Human Data • United States

Remote
USD 55,000 - 96,000
Prior experience training
AI workflow tooling familiarity
Bilingual experience for cross-region