Remote Writer Evaluation & Feedback Specialist

AI Trainer Jobs; United States; Remote; USD 28,000 - 55,000; Part time

Job summary

AuraOne seeks a contractor to join a remote evaluation track for reviewing writer evaluation prompts and responses against its quality rubric. You will compare outputs, label edge cases, and write structured feedback for model retraining.

Responsibilities include evaluating model outputs, tagging issues, and calibrating reviewer quality weekly. Experience with linguistic or trust & safety review is valued, with flexible, async hours from 10 hours per week.

Qualifications

• Prior evaluation, annotation, or human-rater experience on writer evaluation or adjacent content for Writer work.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and rubric clause being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate writer evaluation model outputs against a versioned rubric and assign severity tags for Writer assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.

Skills

Writer evaluation

Rubric-based annotation

Inter-rater calibration

Attention to detail

Clear written reasoning

Job description

AuraOne seeks a contractor to join a remote evaluation track for reviewing writer evaluation prompts and responses against its quality rubric. You will compare outputs, label edge cases, and write structured feedback for model retraining.

Responsibilities include evaluating model outputs, tagging issues, and calibrating reviewer quality weekly. Experience with linguistic or trust & safety review is valued, with flexible, async hours from 10 hours per week.