Remote Rater Calibration QA Reviewer

AuraOne, Inc.; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AuraOne, Inc. seeks a remote Rater Calibration Preference Data Reviewer to evaluate data evaluation prompts and responses against AuraOne's quality rubric. You will compare paired outputs, label edge cases, and provide structured feedback that helps retrain the model.

This contractor role requires strong attention to detail, ability to apply rubrics across long batches, and clear written reasoning. You will work asynchronously for at least 10 hours weekly from the US, with compensation discussed

Qualifications

• Prior evaluation, annotation, or human-rater experience on data evaluation tasks.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning citing rubric clauses.
• Reliable asynchronous availability for at least 10 hours per week.

Responsibilities

• Evaluate model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Document recurring failure modes to guide future rubrics.

Skills

Model evaluation

Rubric annotation

Severity tagging

Inter-rater calibration

Rater Calibration Data evaluation

Preference ranking

RLHF

Job description

AuraOne, Inc. seeks a remote Rater Calibration Preference Data Reviewer to evaluate data evaluation prompts and responses against AuraOne's quality rubric. You will compare paired outputs, label edge cases, and provide structured feedback that helps retrain the model.

This contractor role requires strong attention to detail, ability to apply rubrics across long batches, and clear written reasoning. You will work asynchronously for at least 10 hours weekly from the US, with compensation discussed