Remote Evaluation Set Curator - Data & Policy QA

AuraOne; United States; On-site; USD 34,440 - 55,104; Part time

Job summary

AuraOne is seeking a remote Evaluation Set Curation Data Specialist to assess AI outputs within evaluation workflows. You will review tasks against playbooks, policy, and team guidance to determine if work meets standards.

You will apply structured rubrics, flag policy gaps, and document the next steps for the modeling team, while maintaining consistent inter-rater calibration across batches.

Qualifications

• Direct working experience on real teams evaluating evaluation set curation tasks.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning that cites the policy or workflow used.
• Strong attention to detail and ability to flag problems in prompts.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Review AI outputs against existing evaluation set curation workflows and policies.
• Grade tone, escalation logic, and stakeholder fit on a rubric.
• Flag operational risk, missed escalations, and policy-adherence gaps with severity tags.
• Capture the right next step to train the modeling team.
• Adjudicate disputed playbook calls using firm guidance.
• Maintain reviewer-quality scores in inter-rater calibration cycles.

Skills

Evaluation set curation operations

Rubric application

Policy adherence

Attention to detail

Asynchronous work

Remote contractor work

Job description

AuraOne is seeking a remote Evaluation Set Curation Data Specialist to assess AI outputs within evaluation workflows. You will review tasks against playbooks, policy, and team guidance to determine if work meets standards.

You will apply structured rubrics, flag policy gaps, and document the next steps for the modeling team, while maintaining consistent inter-rater calibration across batches.