Remote Language & Audio AI Evaluator

AI Trainer Jobs; United States; Remote; USD 48,000 - 69,000; Full time

Job summary

AuraOne invites applicants to join its Language & Audio Expert program. This remote contractor role focuses on evaluating audio annotation prompts and responses against a structured rubric. Reviewers assess model outputs, label edge cases, and generate feedback to retrain the system.

You will judge voice agents and pronunciation quality, contributing to robust data labeling practices. Ideal candidates bring linguistics or content-moderation background and experience applying rubrics across long

Qualifications

• Experience evaluating audio annotation or adjacent content.
• Ability to apply multi-page rubrics consistently.
• Clear written reasoning assigning issues to rubric clauses.

Responsibilities

• Evaluate audio annotation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with rationale.
• Label hallucinations, instruction-following failures, and unsafe content with tags.
• Capture ambiguous prompts and route to rubric updates.
• Calibrate reviewer scores against gold-standard examples weekly.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Audio annotation

Job description

AuraOne invites applicants to join its Language & Audio Expert program. This remote contractor role focuses on evaluating audio annotation prompts and responses against a structured rubric. Reviewers assess model outputs, label edge cases, and generate feedback to retrain the system.

You will judge voice agents and pronunciation quality, contributing to robust data labeling practices. Ideal candidates bring linguistics or content-moderation background and experience applying rubrics across long