Remote Workflow Review Specialist – AI Evaluation

AuraOne Human Data

United States

Remote

USD 34,000 - 55,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Remote — US-eligible
Independent contractor

Job summary

AuraOne Human Data is seeking a Generalist (United Kingdom) — Workflow Review Specialist. This remote evaluation track involves reviewing generalist prompts and model outputs against AuraOne's quality rubric, labeling edge cases, and drafting structured feedback for retraining.

Reviewers compare paired outputs, assign severity tags, and document recurring failure modes to improve the rubric and training cycles. Independent contractor role with US-eligible remote work.

Qualifications

  • Prior evaluation, annotation, or human-rater experience on generalist evaluation or adjacent content for Generalist (United Kingdom) — Workflow Review Specialist work.
  • Clear written reasoning that names the issue and the rubric clause being applied.
  • Strong attention to detail and the ability to flag when a prompt itself is the problem.
  • Reliable async availability for at least 10 hours per week.

Responsibilities

  • Evaluate generalist evaluation model outputs against a versioned rubric and assign severity tags for Generalist (United Kingdom) — Workflow Review Specialist assignments.
  • Compare paired responses and pick the stronger answer with a written rationale.
  • Label hallucinations, instruction-following failures, and unsafe content with structured tags.
  • Capture ambiguous prompts and route them back to the program team for rubric updates.
  • Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
  • Document recurring failure modes so the modeling team can target them in the next training run.

Skills

Model output evaluation
Rubric-based annotation
Severity tagging
Inter-rater calibration
Generalist evaluation

Job description

AuraOne Human Data is seeking a Generalist (United Kingdom) — Workflow Review Specialist. This remote evaluation track involves reviewing generalist prompts and model outputs against AuraOne's quality rubric, labeling edge cases, and drafting structured feedback for retraining.

Reviewers compare paired outputs, assign severity tags, and document recurring failure modes to improve the rubric and training cycles. Independent contractor role with US-eligible remote work.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Generalist AI Reviewer - Workflow Quality (Remote Germany)
Generalist AI Reviewer - Workflow Quality (Remote Germany)

AuraOne Human Data • United States

Remote
USD 28,000 - 41,000
Remote AI Workflow Quality Review - Generalist Contractor
Remote AI Workflow Quality Review - Generalist Contractor

AuraOne Human Data • United States

Remote
USD 30,000 - 47,000
Remote Human Data QA & Evaluation Reviewer
Remote Human Data QA & Evaluation Reviewer

Auraone • United States

Remote
USD 34,000 - 55,000
Remote work
Contractor role
Flexible hours
Generalist (United Kingdom) — Workflow Review Specialist
Generalist (United Kingdom) — Workflow Review Specialist

AuraOne Human Data • United States

Remote
USD 34,000 - 55,000
Remote — US-eligible
Independent contractor
Remote Travel Operations AI Workflow Reviewer
Remote Travel Operations AI Workflow Reviewer

AuraOne Human Data • United States

Remote
USD 34,000 - 83,000
Remote Retail AI Workflow Reviewer & Ops QA
Remote Retail AI Workflow Reviewer & Ops QA

AuraOne Human Data • United States

Remote
USD 28,000 - 55,000
Remote Generalist AI Evaluator & Rubric Auditor
Remote Generalist AI Evaluator & Rubric Auditor

AuraOne Human Data • United States

Remote
USD 28,000 - 55,000
Remote AI Data Annotation Evaluator & Quality Reviewer
Remote AI Data Annotation Evaluator & Quality Reviewer

AuraOne, Inc. • United States

Remote
USD 28,000 - 41,000
Remote AI Humanities Evaluation Specialist
Remote AI Humanities Evaluation Specialist

Auraone • United States

Remote
USD 96,000 - 138,000
AI Evaluation Specialist - Remote (US)
AI Evaluation Specialist - Remote (US)

AuraOne Human Data • United States

Remote
USD 28,000 - 48,000
Remote work (US-eligible)
Independent contractor