AI Evaluation Specialist - Remote (US)

AuraOne Human Data

United States

Remote

USD 28,000 - 48,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote work (US-eligible)
Independent contractor

Job summary

AuraOne, Inc. is seeking a Generalist (US Only) reviewer to evaluate model outputs, compare paired responses, and provide structured feedback for retraining. You will apply a versioned rubric, label edge cases, and document recurring issues to help improve the evaluation process.

The role is remote for US-eligible contractors with flexible hours and a minimum of 10 hours per week. Prior experience in evaluation or annotation is preferred, with strong written reasoning and attention to detail.

Qualifications

  • Prior evaluation, annotation, or human-rater experience on generalist evaluation or adjacent content for Generalist (US Only) work.
  • Comfort applying multi-page rubrics consistently across long batches.
  • Clear written reasoning that names the issue and the rubric clause being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Responsibilities

  • Evaluate generalist evaluation model outputs against a versioned rubric and assign severity tags for Generalist (US Only) assignments.
  • Compare paired responses and pick the stronger answer with a written rationale.
  • Label hallucinations, instruction-following failures, and unsafe content with structured tags.
  • Capture ambiguous prompts and route them back to the program team for rubric updates.
  • Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
  • Document recurring failure modes so the modeling team can target them in the next training run.

Job description

AuraOne, Inc. is seeking a Generalist (US Only) reviewer to evaluate model outputs, compare paired responses, and provide structured feedback for retraining. You will apply a versioned rubric, label edge cases, and document recurring issues to help improve the evaluation process.

The role is remote for US-eligible contractors with flexible hours and a minimum of 10 hours per week. Prior experience in evaluation or annotation is preferred, with strong written reasoning and attention to detail.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Evaluation Specialist — Remote (US)
Senior AI Evaluation Specialist — Remote (US)

AuraOne Human Data • United States

Remote
USD 28,000 - 50,000
Remote Generalist AI Evaluator & Rubric Auditor
Remote Generalist AI Evaluator & Rubric Auditor

AuraOne Human Data • United States

Remote
USD 28,000 - 55,000
Remote Evaluation Specialist - Search Generalist
Remote Evaluation Specialist - Search Generalist

AuraOne, Inc. • United States

Remote
USD 41,000 - 55,000
Remote AI Evaluation Rubric Reviewer & Feedback Analyst
Remote AI Evaluation Rubric Reviewer & Feedback Analyst

AuraOne Human Data • United States

Remote
USD 55,000 - 103,000
Remote AI Evaluation & Rubric Annotation Specialist
Remote AI Evaluation & Rubric Annotation Specialist

AuraOne Human Data • United States

Remote
USD 28,000 - 48,000
Remote AI Evaluation Rubric Reviewer & Feedback Specialist
Remote AI Evaluation Rubric Reviewer & Feedback Specialist

AuraOne Human Data • United States

Remote
USD 34,000 - 55,000
AI Evaluation Data Analyst — Remotely Review & Calibrate
AI Evaluation Data Analyst — Remotely Review & Calibrate

Auraone • United States

Remote
USD 97,000 - 138,000
Remote AI Search Evaluation Specialist
Remote AI Search Evaluation Specialist

AuraOne Human Data • United States

Remote
USD 41,000 - 55,000
AI Training Evaluator & Rubric Analyst — Remote Contractor
AI Training Evaluator & Rubric Analyst — Remote Contractor

Auraone • United States

Remote
USD 34,000 - 55,000
Remote: Japanese AI Evaluation & Feedback Specialist
Remote: Japanese AI Evaluation & Feedback Specialist

Auraone • United States

Remote
USD 58,000 - 80,000