Remote RLHF: Reward Hacking Preference Data Reviewer

AI Trainer Jobs; United States; Remote; USD 27,552,000 - 41,328,000; Full time

Benefits offered by this job

Remote work (US-eligible)

Contractor role

Job summary

AuraOne is seeking a Reward Hacking Detection Preference Data Reviewer for a remote evaluation track. You will review data prompts and responses against our quality rubric, compare paired outputs, and provide structured feedback to retrain models.

Responsibilities include labeling issues, ranking outputs, and writing clear rationales. This is a contractor, remote role with hourly pay to be confirmed after interview, and US eligibility is required.

Qualifications

• Responsibilities include evaluating model outputs against a rubric, labeling issues, and providing feedback.
• Ability to apply rubric criteria consistently to long text prompts.
• Strong written reasoning and attention to detail.
• Experience with RLHF, human preference data evaluation, or content moderation is a plus.

Responsibilities

• Evaluate reward hacking detection preference data evaluation model outputs against a versioned rubric and assign severity tags for Reviewer assignments.
• Compare paired responses and select the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.

Job description

AuraOne is seeking a Reward Hacking Detection Preference Data Reviewer for a remote evaluation track. You will review data prompts and responses against our quality rubric, compare paired outputs, and provide structured feedback to retrain models.

Responsibilities include labeling issues, ranking outputs, and writing clear rationales. This is a contractor, remote role with hourly pay to be confirmed after interview, and US eligibility is required.