Remote Evaluation QA & Ontology Curation Reviewer

AuraOne; United States; On-site; USD 55,104 - 82,656; Part time

Job summary

AuraOne is seeking a remote Evaluation Set Curation Ontology QA Reviewer to assess evaluation outputs against a quality rubric. You will compare paired responses, assign severity tags, and provide structured feedback to aid retraining.

This role is a contractor, remote for US-based work, requiring reliable weekly availability. Responsibilities include labeling issues, routing ambiguous prompts for rubric updates, and maintaining calibration against gold-standard examples.

Qualifications

• Prior evaluation, annotation, or human-rater experience on evaluation set curation ontology QA evaluation or adjacent content.
• Clear written reasoning that names the issue and rubric clause.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Calibrate reviewer quality against gold-standard examples weekly.
• Document recurring failure modes for future training runs.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Data quality

Ontology review

Curation/Evaluation

Job description

AuraOne is seeking a remote Evaluation Set Curation Ontology QA Reviewer to assess evaluation outputs against a quality rubric. You will compare paired responses, assign severity tags, and provide structured feedback to aid retraining.

This role is a contractor, remote for US-based work, requiring reliable weekly availability. Responsibilities include labeling issues, routing ambiguous prompts for rubric updates, and maintaining calibration against gold-standard examples.