Remote AI Evaluation Rubric Reviewer & Feedback Analyst

AuraOne Human Data

United States

Remote

USD 55,000 - 103,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

AuraOne, Inc. is seeking an AI Evaluation Rubric Reviewer for a remote contractor role. You will assess model outputs against AuraOne's rubric, label edge cases, and provide structured feedback for retraining.

Responsibilities include evaluating rubric outputs, tagging issues, calibrating scores, and documenting recurring failure modes. Must have prior evaluation or annotation experience and be available at least 10 hours per week.

Qualifications

  • Experience evaluating or annotating AI evaluation rubrics.
  • Ability to apply rubrics consistently across long batches.
  • Clear written reasoning naming issues and rubric clauses.
  • Attention to detail and ability to flag prompt problems.
  • Reliable async availability for at least 10 hours per week.

Responsibilities

  • Evaluate rubric outputs and assign severity tags.
  • Compare paired responses and document rationale.
  • Label hallucinations and unsafe content with policy categories.
  • Capture ambiguous prompts for rubric updates.
  • Calibrate reviewer scores against gold-standard examples.
  • Document recurring failure modes for future training.

Skills

Model output evaluation
Rubric-based annotation
Severity tagging
Inter-rater calibration
AI Evaluation Rubric evaluation

Job description

AuraOne, Inc. is seeking an AI Evaluation Rubric Reviewer for a remote contractor role. You will assess model outputs against AuraOne's rubric, label edge cases, and provide structured feedback for retraining.

Responsibilities include evaluating rubric outputs, tagging issues, calibrating scores, and documenting recurring failure modes. Must have prior evaluation or annotation experience and be available at least 10 hours per week.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Evaluation Rubric Reviewer & Feedback Specialist
Remote AI Evaluation Rubric Reviewer & Feedback Specialist

AuraOne Human Data • United States

Remote
USD 34,000 - 55,000
Remote AI Image Quality Reviewer & Rubric Specialist
Remote AI Image Quality Reviewer & Rubric Specialist

AuraOne Human Data • United States

Remote
USD 28,000 - 48,000
Remote AI Evaluation & Feedback Specialist
Remote AI Evaluation & Feedback Specialist

Auraone • United States

Remote
USD 34,000 - 83,000
Remote work
Contractor engagement
US-eligible
Remote AI Product Evaluator — Rubric-Based Reviews
Remote AI Product Evaluator — Rubric-Based Reviews

AuraOne, Inc. • United States

Remote
USD 34,000 - 48,000
Remote work within US
Remote Rubric Evaluation Specialist
Remote Rubric Evaluation Specialist

Auraone • United States

Remote
USD 28,000 - 55,000
AI Training Evaluator & Rubric Analyst — Remote Contractor
AI Training Evaluator & Rubric Analyst — Remote Contractor

Auraone • United States

Remote
USD 34,000 - 55,000
Remote AI Evaluation & Annotation Specialist
Remote AI Evaluation & Annotation Specialist

Auraone • United States

Remote
USD 28,000 - 55,000
Remote AI Evaluation Specialist - Task-Completion Quality
Remote AI Evaluation Specialist - Task-Completion Quality

AuraOne Human Data • United States

Remote
USD 34,000 - 62,000
AI Agent Evaluator — Remote Rubric & Feedback Specialist
AI Agent Evaluator — Remote Rubric & Feedback Specialist

Auraone • United States

Remote
USD 34,000 - 55,000
Remote AI Data Annotation Evaluator & Quality Reviewer
Remote AI Data Annotation Evaluator & Quality Reviewer

AuraOne, Inc. • United States

Remote
USD 28,000 - 41,000