Remote Business Intelligence AI Evaluator

AuraOne; United States; On-site; USD 27,552 - 55,104; Full time

Job summary

AuraOne is seeking a remote Independent Contractor to review BI AI evaluation prompts and responses. You will compare outputs to a versioned rubric, label issues, and provide structured feedback for retraining.

The role emphasizes careful reasoning, consistent rubric application, and documenting edge cases for model improvement. The work is remote, US-eligible, with an hourly compensation discussed after interview.

Qualifications

• Experience evaluating AI outputs or related data labeling tasks.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning naming issues and rubric clauses.

Responsibilities

• Evaluate BI AI evaluation outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and flag for rubric updates.
• Maintain reviewer quality scores through calibration against gold-standard samples.
• Document recurring failure modes to guide model retraining.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Workflow evaluation

Operational judgment

Business Intelligence

Task prioritization

Tools

None

Job description

AuraOne is seeking a remote Independent Contractor to review BI AI evaluation prompts and responses. You will compare outputs to a versioned rubric, label issues, and provide structured feedback for retraining.

The role emphasizes careful reasoning, consistent rubric application, and documenting edge cases for model improvement. The work is remote, US-eligible, with an hourly compensation discussed after interview.