AI Agent Evaluator — Remote Rubric & Feedback Specialist

AuraOne; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining.

The role focuses on auditing model outputs, labeling issues with rubric clauses, and calibrating scores weekly to maintain high reviewer quality. Applicants must be authorized to work remotely in the US.

Qualifications

• Experience evaluating AI agent outputs against rubrics.
• Ability to apply rubrics consistently across long batches.
• Clear written reasoning naming issues and rubric clauses.
• Attention to detail and ability to flag problematic prompts.

Responsibilities

• Evaluate AI agent evaluation outputs against a rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations and unsafe content with structured tags.
• Capture ambiguous prompts and route to rubric updates.
• Maintain reviewer quality scores by calibrating against gold-standard examples.
• Document recurring failure modes for model training.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

AI Agent evaluation

AI model evaluation

Job description

AuraOne is seeking an AI Agent Evaluator to remotely review and label ai agent evaluation prompts and responses against AuraOne's quality rubric. You will compare outputs, identify edge cases, and provide structured feedback to support retraining.

The role focuses on auditing model outputs, labeling issues with rubric clauses, and calibrating scores weekly to maintain high reviewer quality. Applicants must be authorized to work remotely in the US.