Remote AI Search Evaluation Specialist (Generalist)

AuraOne Human Data; United States; Remote; USD 41.000 - 55.000; Teilzeit

Zusammenfassung

AuraOne Human Data seeks remote reviewers to evaluate AI search generalist outputs, applying a versioned rubric and labeling edge cases. You will compare responses, justify choices, and help calibrate quality with weekly gold-standard checks.

Ideal candidates have prior evaluation or annotation experience, strong written reasoning, and reliable async availability of 10 hours per week. Remote, contractor-based engagement with variable volume.

Qualifikationen

• Prior evaluation, annotation, or human-rater experience on search generalist evaluation or adjacent content.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Strong attention to detail and the ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Aufgaben

• Evaluate search generalist evaluation model outputs against a versioned rubric and assign severity tags for Search Generalist — AI Search Evaluation (Remote) assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.
• Document recurring failure modes so the modeling team can target them in the next training run.

Kenntnisse

Evaluation experience

Annotation

Human-rater experience

Rubric application

Written reasoning

Attention to detail

Async availability

Linguistics background

Jobbeschreibung

AuraOne Human Data seeks remote reviewers to evaluate AI search generalist outputs, applying a versioned rubric and labeling edge cases. You will compare responses, justify choices, and help calibrate quality with weekly gold-standard checks.

Ideal candidates have prior evaluation or annotation experience, strong written reasoning, and reliable async availability of 10 hours per week. Remote, contractor-based engagement with variable volume.