Remote Prompt Robustness Reviewer and Feedback Analyst

AuraOne, Inc.; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

AuraOne, Inc. seeks a remote Prompt Robustness Failure Analysis Reviewer to evaluate model outputs against a versioned rubric and assign severity tags. You will compare paired responses, justify selections, and label issues using structured rubric categories.

The role emphasizes clear written reasoning, multi-page rubric consistency, and calibration against gold-standard examples. Independent contractor arrangement with US remote work. Hourly compensation discussed after interview.

Qualifications

• Experience evaluating prompts and annotating results using defined rubrics.
• Ability to apply rubric clauses consistently across long batches.
• Clear written reasoning that names issues and rubric references.

Responsibilities

• Evaluate prompt robustness failure analysis model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with a rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them for rubric updates.
• Maintain reviewer-quality scores by calibration against gold-standard samples.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Prompt robustness

Frontier evaluation

Rubric calibration

Failure analysis

Attention to detail

Clear written reasoning

Job description

AuraOne, Inc. seeks a remote Prompt Robustness Failure Analysis Reviewer to evaluate model outputs against a versioned rubric and assign severity tags. You will compare paired responses, justify selections, and label issues using structured rubric categories.

The role emphasizes clear written reasoning, multi-page rubric consistency, and calibration against gold-standard examples. Independent contractor arrangement with US remote work. Hourly compensation discussed after interview.