Arabic Evaluation Specialist - Model Feedback (Remote)

AI Trainer Jobs; United States; Remote; USD 22,000 - 25,000; Full time

Job summary

AuraOne seeks an Arabic expert for a remote evaluation track to review Arabic generalist prompts and responses against our quality rubric. Reviewers compare paired outputs, tag edge cases, and write structured feedback for retraining.

As a contractor, you will help turn outputs into auditable labels, rationales, and regression cases, calibrate ratings against gold standards weekly, and participate in rubric updates. Reliable async availability for at least 10 hours per week is required.

Qualifications

• Prior evaluation, annotation, or human-rater experience on arabic generalist evaluation or adjacent content.
• Comfort applying multi-page rubrics consistently across long batches.
• Clear written reasoning that names the issue and the rubric clause being applied.
• Reliable async availability for at least 10 hours per week.

Responsibilities

• Evaluate arabic generalist evaluation model outputs against a versioned rubric and assign severity tags for Arabic expert assignments.
• Compare paired responses and pick the stronger answer with a written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and route them back to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples each week.

Skills

Arabic generalist evaluation

Inter-rater calibration

Severity tagging

Model output evaluation

Rubric-based annotation

Job description

AuraOne seeks an Arabic expert for a remote evaluation track to review Arabic generalist prompts and responses against our quality rubric. Reviewers compare paired outputs, tag edge cases, and write structured feedback for retraining.

As a contractor, you will help turn outputs into auditable labels, rationales, and regression cases, calibrate ratings against gold standards weekly, and participate in rubric updates. Reliable async availability for at least 10 hours per week is required.