Remote Agent Regression QA & Evaluation Specialist

AuraOne; United States; On-site; USD 34,440 - 82,656; Part time

Job summary

AuraOne is seeking an Agent Regression Test Evaluation Specialist to remotely review AI outputs and ensure workflow, policy, and stakeholder alignment. This contractor role involves grading tone, escalation logic, and adherence to playbooks, then documenting the next steps for training the modeling team.

The position requires clear written reasoning, strong attention to detail, and reliable async availability for at least ten hours weekly. Experience with multi-page rubrics is a plus.

Qualifications

• Direct working experience in agent regression test evaluation specialist operations on real teams.
• Clear written reasoning that names the policy or workflow being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Review AI outputs against current agent regression test evaluation specialist operations workflows, playbooks, and firm policy for assignments.
• Grade tone, escalation logic, and stakeholder fit on a structured rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputed treatments against published playbooks or firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Operational review

Policy adherence

Workflow judgment

Stakeholder communication

Agent regression testing

Code review

Software testing

Agent evaluation

Regression testing

Test

Job description

AuraOne is seeking an Agent Regression Test Evaluation Specialist to remotely review AI outputs and ensure workflow, policy, and stakeholder alignment. This contractor role involves grading tone, escalation logic, and adherence to playbooks, then documenting the next steps for training the modeling team.

The position requires clear written reasoning, strong attention to detail, and reliable async availability for at least ten hours weekly. Experience with multi-page rubrics is a plus.