Remote Video Understanding Evaluator & Rubric Auditor

AuraOne; United States; On-site; USD 41,000 - 69,000; Part time

Benefits offered by this job

Remote work

Job summary

AuraOne is seeking a remote Video Understanding AI Evaluator contractor to assess model outputs against a quality rubric and provide detailed rationale. You will compare paired responses, label edge cases, and report ambiguous prompts for rubric updates.

The role focuses on maintaining high reviewer quality and documenting failure modes for retraining the modeling team. Applicants should have prior experience in evaluation/annotation of AI content, meticulous attention to detail, and reliable

Qualifications

• Experience evaluating video understanding outputs or adjacent content.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning naming the issue and rubric clause.
• Strong attention to detail and flagging prompt problems.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate video evaluation model outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label unsafe or problematic content with structured tags.
• Capture ambiguous prompts and route for rubric updates.
• Calibrate reviewer quality against gold-standard examples weekly.
• Document recurring failure modes for future training.

Skills

Video evaluation experience

Written rationale

Attention to detail

Async availability

Inter-rater calibration

Tools

Annotation tools

Job description

AuraOne is seeking a remote Video Understanding AI Evaluator contractor to assess model outputs against a quality rubric and provide detailed rationale. You will compare paired responses, label edge cases, and report ambiguous prompts for rubric updates.

The role focuses on maintaining high reviewer quality and documenting failure modes for retraining the modeling team. Applicants should have prior experience in evaluation/annotation of AI content, meticulous attention to detail, and reliable