Remote AI Product Evaluator — Rubric-Based Reviews

AuraOne, Inc.; United States; Remote; USD 34,000 - 48,000; Part time

Benefits offered by this job

Remote work within US

Job summary

AuraOne, Inc. is seeking an AI Product Evaluator for a remote contractor role. You will review prompts and model outputs, apply a versioned rubric, and provide structured feedback to retrain the model.

The work involves tagging issues, labeling edge cases, and maintaining calibration with gold-standard examples each week. The role emphasizes clear written reasoning, attention to detail, and async availability of at least 10 hours per week.

Qualifications

• Prior evaluation, annotation, or human-rater experience in AI product evaluation or related content.
• Clear written reasoning that explains the issue and rubric clause.
• Attention to detail and ability to flag prompt issues.

Responsibilities

• Evaluate AI product evaluation outputs against a rubric and assign severity tags.
• Compare paired responses and write a rational justification for the stronger answer.
• Label hallucinations, policy violations, and safety issues with structured tags.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

AI Product evaluation

AI model evaluation

Job description

AuraOne, Inc. is seeking an AI Product Evaluator for a remote contractor role. You will review prompts and model outputs, apply a versioned rubric, and provide structured feedback to retrain the model.

The work involves tagging issues, labeling edge cases, and maintaining calibration with gold-standard examples each week. The role emphasizes clear written reasoning, attention to detail, and async availability of at least 10 hours per week.