Remote AI Conversation Quality Auditor

AuraOne Human Data

United States

Remote

USD 34,000 - 55,000

Part time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Linguistics background
Inter-rater metrics experience
Subject-matter domain knowledge

Job summary

AuraOne, Inc. seeks an AI Conversation Evaluator (Remote) to review prompts and responses against our quality rubric. You will compare paired outputs, label edge cases, and provide structured feedback for model retraining.

This contractor role requires clear written reasoning, meticulous attention to detail, and reliable async availability for at least 10 hours per week. US residency is required; compensation is hourly and determined after interview.

Qualifications

  • Must have prior evaluation, annotation, or human-rater experience on ai conversation evaluation or adjacent content.
  • Comfort applying multi-page rubrics consistently across long batches.
  • Clear written reasoning that names the issue and the rubric clause being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Responsibilities

  • Evaluate AI conversation evaluation model outputs against a rubric and assign severity tags.
  • Compare paired responses and select the stronger answer with a rationale.
  • Label hallucinations, policy violations, and unsafe content with structured tags.
  • Capture ambiguous prompts and route them for rubric updates.
  • Calibrate reviewer scores against gold-standard examples each week.
  • Document recurring failure modes for model improvement.

Skills

Prior evaluation/annotation experience
Apply multi-page rubrics
Clear written reasoning
Attention to detail
Async availability

Job description

AuraOne, Inc. seeks an AI Conversation Evaluator (Remote) to review prompts and responses against our quality rubric. You will compare paired outputs, label edge cases, and provide structured feedback for model retraining.

This contractor role requires clear written reasoning, meticulous attention to detail, and reliable async availability for at least 10 hours per week. US residency is required; compensation is hourly and determined after interview.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Remote AI Chat Quality Reviewer (Contractor)
Remote AI Chat Quality Reviewer (Contractor)

AuraOne Human Data • United States

Remote
USD 34,000 - 49,000
Remote work
US-eligible contractor
Remote AI Evaluation & Feedback Specialist
Remote AI Evaluation & Feedback Specialist

Auraone • United States

Remote
USD 34,000 - 83,000
Remote work
Contractor engagement
US-eligible
Remote AI Evaluation & Annotation Specialist
Remote AI Evaluation & Annotation Specialist

Auraone • United States

Remote
USD 28,000 - 55,000
AI Conversation Evaluator (Remote)
AI Conversation Evaluator (Remote)

AuraOne Human Data • United States

Remote
USD 34,000 - 55,000
Linguistics background
Inter-rater metrics experience
Subject-matter domain knowledge
Remote AI Conversation Quality Analyst
Remote AI Conversation Quality Analyst

Alignerr • United States

Remote
USD 25,000 - 41,000
Remote AI Evaluation Specialist - Task-Completion Quality
Remote AI Evaluation Specialist - Task-Completion Quality

AuraOne Human Data • United States

Remote
USD 34,000 - 62,000
Remote Social Content Quality Auditor
Remote Social Content Quality Auditor

AuraOne, Inc. • United States

Remote
USD 34,000 - 55,000
Remote work
Remote Historical Reasoning AI Evaluator & Feedback
Remote Historical Reasoning AI Evaluator & Feedback

AuraOne, Inc. • United States

Remote
USD 34,000 - 62,000
Remote work
Contractor role
Remote Video AI Evaluation Specialist
Remote Video AI Evaluation Specialist

Auraone • United States

Remote
USD 34,000 - 55,000
Remote AI Executive Communications QA Reviewer
Remote AI Executive Communications QA Reviewer

Auraone • United States

Remote
USD 96,000 - 103,000