Remote PR Reasoning Evaluator | AI Workflow QA

AuraOne; United States; On-site; USD 55,104 - 96,432; Part time

Job summary

AuraOne is seeking a Pull Request Reasoning Evaluation Specialist to remotely review AI outputs across pull request workflows, playbooks, and firm policy, grading correctness and stakeholder fit.

You will flag operational risk, capture the right next steps for model training, and adjudicate disputes using published guidance while maintaining inter-rater calibration scores.

This is a contractor role, US-eligible, with compensation discussed after interview.

Qualifications

• Experience on real PR reasoning evaluation operations teams.
• Ability to apply multi-page rubrics consistently.
• Clear written reasoning citing policy/workflow.
• Excellent attention to detail and ability to flag issues.
• Available asynchronously for at least 10 hours per week.

Responsibilities

• Review AI outputs against current PR reasoning evaluation workflows, playbooks, and firm policy for assignments.
• Grade tone, escalation logic, and stakeholder fit on a structured rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputed playbooks or guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Direct PR evaluation experience

Rubric application

Policy/workflow justification

Attention to detail

Part-time availability

Job description

AuraOne is seeking a Pull Request Reasoning Evaluation Specialist to remotely review AI outputs across pull request workflows, playbooks, and firm policy, grading correctness and stakeholder fit.

You will flag operational risk, capture the right next steps for model training, and adjudicate disputes using published guidance while maintaining inter-rater calibration scores.

This is a contractor role, US-eligible, with compensation discussed after interview.