Remote Metadata Ontology QA Evaluator

AuraOne; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AuraOne is seeking a remote Metadata Normalization Ontology QA Reviewer to evaluate model outputs against a versioned rubric. You will label edge cases, write structured feedback, and help retrain the modeling team.

This contract role requires reliable async availability and attention to detail. You will compare paired responses, tag inaccuracies, and document recurring failures to improve data quality for AuraOne Human Data.

Qualifications

• Prior evaluation/annotation experience on metadata normalization ontology QA evaluation or similar tasks.
• Ability to apply rubrics consistently across long batches.
• Clear written reasoning that names the issue and rubric clause.
• Attention to detail and reliability in asynchronous work.

Responsibilities

• Evaluate metadata normalization ontology QA outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Calibrate reviewer-quality scores against gold-standard examples weekly.
• Document recurring failure modes for targeted retraining.

Skills

Eval experience

Annotation

Human raters

Metadata QA

Reasoning clarity

Attention to detail

Async availability

Job description

AuraOne is seeking a remote Metadata Normalization Ontology QA Reviewer to evaluate model outputs against a versioned rubric. You will label edge cases, write structured feedback, and help retrain the modeling team.

This contract role requires reliable async availability and attention to detail. You will compare paired responses, tag inaccuracies, and document recurring failures to improve data quality for AuraOne Human Data.