Tagalog AI Evaluation Psychologist (Contractor)

AuraOne, Inc.; United States; Remote; USD 34,000 - 55,000; Part time

Job summary

AuraOne, Inc. is seeking a remote Tagalog Speaking Psychologist (PhD) to evaluate model outputs against a versioned rubric. You will label content, identify errors, and provide structured feedback to train AI systems. This is a CONTRACTOR role with async availability and strict adherence to rubrics.

Responsibilities include comparing responses, tagging unsafe content, and calibrating scores against gold-standard examples weekly. Background in linguistics, moderation, or trust & safety is a plus.

Qualifications

• Experience evaluating model outputs using a predefined rubric.
• Ability to annotate with structured rationales and labels.
• Experience applying severity tags to content.
• Ability to calibrate judgments across multiple raters.
• Fluency in Tagalog for evaluating generalist prompts.

Responsibilities

• Evaluate Tagalog generalist evaluation model outputs against a versioned rubric and assign severity tags.
• Compare paired responses and select the stronger answer with written rationale.
• Label hallucinations, instruction-following failures, and unsafe content with structured tags.
• Capture ambiguous prompts and escalate to the program team for rubric updates.
• Maintain reviewer-quality scores by calibrating against gold-standard examples weekly.
• Document recurring failure modes to guide future training.

Skills

Model output evaluation

Rubric-based annotation

Severity tagging

Inter-rater calibration

Tagalog generalist evaluation

Education

PhD in Psychology

Job description

AuraOne, Inc. is seeking a remote Tagalog Speaking Psychologist (PhD) to evaluate model outputs against a versioned rubric. You will label content, identify errors, and provide structured feedback to train AI systems. This is a CONTRACTOR role with async availability and strict adherence to rubrics.

Responsibilities include comparing responses, tagging unsafe content, and calibrating scores against gold-standard examples weekly. Background in linguistics, moderation, or trust & safety is a plus.