Remote CI Failure Workflow Reviewer & Feedback Annotator

AuraOne, Inc.; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AuraOne, Inc. is seeking a remote CI Failure Workflow Reviewer to evaluate model outputs against a versioned rubric, label edge cases, and provide structured feedback for retraining. The role focuses on comparing paired responses, tagging issues, and maintaining calibration over time.

Ideal candidates have prior experience with ci failure workflows, strong written reasoning, and reliable async availability for at least 10 hours per week. This is a contractor, remote US-eligible position.

Qualifications

• Prior evaluation, annotation, or human-rater experience on ci failure workflow evaluation or adjacent content.

Responsibilities

• Evaluate ci failure workflow evaluation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and provide rationale for stronger answer.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples weekly.
• Document recurring failure modes for future training.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

CI Failure Workflow evaluation

Code review

Software testing

Agent evaluation

Failure

Job description

AuraOne, Inc. is seeking a remote CI Failure Workflow Reviewer to evaluate model outputs against a versioned rubric, label edge cases, and provide structured feedback for retraining. The role focuses on comparing paired responses, tagging issues, and maintaining calibration over time.

Ideal candidates have prior experience with ci failure workflows, strong written reasoning, and reliable async availability for at least 10 hours per week. This is a contractor, remote US-eligible position.