Remote: Academic Search Task Evaluator & Quality Labeler

AuraOne; United States; On-site; USD 34,440 - 55,104; Full time

Job summary

AuraOne is seeking an Academic Search Task Evaluator to remotely review evaluation prompts and responses against AuraOne's quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for retraining by the modeling team.

Applicants should have prior evaluation or annotation experience, strong attention to detail, and be able to apply rubrics consistently across long batches. Remote contractor role with US eligibility.

Qualifications

• Prior evaluation, annotation, or human-rater experience on academic search task evaluation or adjacent content for Academic Search Task Evaluator work.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples weekly.
• Document recurring failure modes for model retraining.

Skills

Model evaluation

Rubric-based labeling

Severity tagging

Inter-rater calibration

Academic Search Task evaluation

Web research

Tools

Browser automation

Source grounding

Web research

Job description

AuraOne is seeking an Academic Search Task Evaluator to remotely review evaluation prompts and responses against AuraOne's quality rubric. You will compare paired outputs, label edge cases, and write structured feedback for retraining by the modeling team.

Applicants should have prior evaluation or annotation experience, strong attention to detail, and be able to apply rubrics consistently across long batches. Remote contractor role with US eligibility.