Remote AI Workflow Evaluator – Management Consultant

AuraOne; United States; On-site; USD 34,000 - 83,000; Part time

Job summary

AuraOne is seeking a remote contractor to evaluate management consultant AI workflow evaluation outputs against a versioned rubric. You will compare paired prompts, assign severity tags, and provide structured feedback to help retrain the modeling team.

Responsibilities include tagging hallucinations, outlining rationales, and documenting recurring failure modes. The role requires clear reasoning and strict adherence to rubric clauses, with about 10 hours per week remote availability.

Qualifications

• Experience with evaluation, annotation, or human-rater work in AI workflow evaluation or related content.
• Ability to apply multi-page rubrics consistently across large batches.
• Clear written reasoning that identifies issues and cites rubric clauses.
• Strong attention to detail and ability to flag prompt issues.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate AI workflow evaluation model outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content using structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibration against gold-standard examples.
• Document recurring failure modes for future training.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Management Consultant AI Workflow eval

Enterprise operations

AI evaluation

Rubric writing

Expert review

Job description

AuraOne is seeking a remote contractor to evaluate management consultant AI workflow evaluation outputs against a versioned rubric. You will compare paired prompts, assign severity tags, and provide structured feedback to help retrain the modeling team.

Responsibilities include tagging hallucinations, outlining rationales, and documenting recurring failure modes. The role requires clear reasoning and strict adherence to rubric clauses, with about 10 hours per week remote availability.