Remote Prompt Robustness Evaluator (Model Evaluation)

AI Trainer Jobs; United States; Remote; USD 34,000 - 62,000; Part time

Job summary

AuraOne is seeking a Prompt Robustness Model Evaluation Specialist to remotely review evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and provide structured feedback to retrain the model.

Responsibilities include evaluating frontier model outputs, judging failures, and calibrating other evaluators. Strong written reasoning and attention to detail are essential, with at least 10 hours per week availability.

Qualifications

• Prior evaluation or annotation experience on prompt robustness evaluation.
• Ability to apply multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and the rubric clause applied.
• Strong attention to detail and ability to flag when a prompt itself is the problem.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate prompt robustness model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.

Skills

Attention to detail

Written reasoning

Analytical thinking

Inter-rater calibration

Linguistics background

Job description

AuraOne is seeking a Prompt Robustness Model Evaluation Specialist to remotely review evaluation prompts and responses against AuraOne's quality rubric. Reviewers compare paired outputs, label edge cases, and provide structured feedback to retrain the model.

Responsibilities include evaluating frontier model outputs, judging failures, and calibrating other evaluators. Strong written reasoning and attention to detail are essential, with at least 10 hours per week availability.