Remote AI Evaluation Specialist - Task-Completion Quality

AuraOne, Inc.; United States; Remote; USD 34,000 - 62,000; Part time

Job summary

AuraOne, Inc. is seeking a remote evaluation specialist to review agentic task completion model outputs, apply a structured rubric, and provide rationales for retraining. The role focuses on tagging edge cases, formatting auditable feedback, and maintaining calibration against gold-standard examples.

Remote contractor position requiring reliable async availability for at least 10 hours per week. Compensation discussed after interviews.

Qualifications

• Experience evaluating model outputs or adjacent content using rubrics.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning naming the issue and rubric clause.

Responsibilities

• Evaluate agentic task completion model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route to program team for rubric updates.
• Calibrate reviewer-quality scores against gold-standard examples weekly.
• Document recurring failure modes for future training runs.

Skills

Model evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Agentic task completion

Failure analysis

Rubric calibration

Job description

AuraOne, Inc. is seeking a remote evaluation specialist to review agentic task completion model outputs, apply a structured rubric, and provide rationales for retraining. The role focuses on tagging edge cases, formatting auditable feedback, and maintaining calibration against gold-standard examples.

Remote contractor position requiring reliable async availability for at least 10 hours per week. Compensation discussed after interviews.