Remote Multi-Turn Ranking Feedback Reviewer

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

AI Trainer Jobs is seeking a Remote Multi-Turn Ranking Preference Data Reviewer to support AuraOne's data quality program. This independent contractor role involves evaluating prompts and responses, labeling edge cases, and providing structured feedback to help retrain models.

You will compare paired outputs, tag issues with rubric references, and help produce regression data. The role is remote, US-eligible, and compensation is hourly and determined after interview.

Qualifications

• Experience evaluating multi-turn ranking data and providing structured feedback.
• Ability apply rubrics consistently across long batches.
• Clear written reasoning naming issues and rubric clauses.

Responsibilities

• Evaluate model outputs against rubric and assign severity tags.
• Compare paired responses and select stronger with rationale.
• Label hallucinations, instruction-following failures, and unsafe content.
• Capture ambiguous prompts and route to rubric updates.

Skills

Rubric-based annotation

Inter-rater calibration

Multi-turn evaluation

RLHF

Job description

AI Trainer Jobs is seeking a Remote Multi-Turn Ranking Preference Data Reviewer to support AuraOne's data quality program. This independent contractor role involves evaluating prompts and responses, labeling edge cases, and providing structured feedback to help retrain models.

You will compare paired outputs, tag issues with rubric references, and help produce regression data. The role is remote, US-eligible, and compensation is hourly and determined after interview.