Remote RLHF Evaluator: Preference Taxonomy Feedback

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AuraOne seeks a remote evaluator to review preference taxonomy reward model prompts and responses against its quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for retraining the modeling team.

As a contractor, you will contribute to auditable labels, rationales, and regression cases, producing preference rankings and calibrated human judgments for post-training pipelines. Prior evaluation or moderation experience is valued.

Qualifications

• Experience in evaluation, annotation, or human-rating for preference models.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning naming the issue and rubric clause.

Responsibilities

• Evaluate model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger with rationale.
• Label hallucinations and unsafe content with structured tags.
• Capture ambiguous prompts for rubric updates.

Skills

Linguistics background

Annotation experience

Inter-rater calibration

Attention to detail

Written reasoning

Job description

AuraOne seeks a remote evaluator to review preference taxonomy reward model prompts and responses against its quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for retraining the modeling team.

As a contractor, you will contribute to auditable labels, rationales, and regression cases, producing preference rankings and calibrated human judgments for post-training pipelines. Prior evaluation or moderation experience is valued.