Senior AI Evaluation Specialist — Remote

AI Trainer Jobs; United States; Remote; USD 34,000 - 55,000; Full time

Job summary

AuraOne is seeking a Senior AI Trainer for a remote evaluation track. Review prompts and model outputs against a quality rubric, compare paired responses, and provide structured feedback to retrain the model.

The role emphasizes labeling edge cases, documenting rationale, and maintaining calibration with gold standards. The position is contractor-based with tasks defined by program volume, available on a remote basis for US-eligible candidates.

Qualifications

• Prior evaluation, annotation, or human-rater experience on Senior AI Trainer work.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning naming the issue and rubric clause being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate senior AI trainer evaluation model outputs against a versioned rubric and assign severity tags for Senior AI Trainer assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.

Skills

Evaluation experience

Annotation experience

Inter-rater calibration

Attention to detail

Job description

AuraOne is seeking a Senior AI Trainer for a remote evaluation track. Review prompts and model outputs against a quality rubric, compare paired responses, and provide structured feedback to retrain the model.

The role emphasizes labeling edge cases, documenting rationale, and maintaining calibration with gold standards. The position is contractor-based with tasks defined by program volume, available on a remote basis for US-eligible candidates.