AI Evaluation Specialist — Recent Grad (Remote)

AuraOne; United States; On-site; USD 55,000 - 83,000; Full time

Job summary

AuraOne is seeking an Evaluation Specialist/Recent Grad contractor to review AI outputs and assess alignment with evaluation workflows, policy, and stakeholder fit. You will document next steps to guide the modeling team in training and improvements.

The role requires attention to detail, the ability to work asynchronously at least 10 hours per week, and comfort applying multi-page rubrics to large batches while clearly naming the policies used.

Qualifications

• Experience with evaluating AI outputs and applying firm policy.
• Ability to apply multi-page rubrics consistently across batches.
• Clear written reasoning naming the policy/workflow applied.

Responsibilities

• Review AI outputs against evaluation workflows, playbooks and firm policy.
• Grade tone, escalation and stakeholder fit on a structured rubric.
• Flag operational risk, missed escalations and policy-adherence gaps with severity tags.
• Capture the right next step so the modeling team can train on it.
• Adjudicate disputes using published playbooks or firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Operational review

Policy adherence

Workflow judgment

Stakeholder communication

Evaluation /Recent Grad specialist ops

Job description

AuraOne is seeking an Evaluation Specialist/Recent Grad contractor to review AI outputs and assess alignment with evaluation workflows, policy, and stakeholder fit. You will document next steps to guide the modeling team in training and improvements.

The role requires attention to detail, the ability to work asynchronously at least 10 hours per week, and comfort applying multi-page rubrics to large batches while clearly naming the policies used.