Remote AI Evaluation Specialist (Live Conversations)

Scale AI; Hamilton; Hybrid; CAD 24,000 - 34,000; Full time

Benefits offered by this job

Flexible schedule

Free access to leading LLMs

Job summary

Outlier is an AI training platform owned and operated by Scale AI. We are seeking detail-oriented English speakers to review AI-generated media, roleplaying live conversations with an AI voice model and scoring each exchange, and explaining what looked or sounded wrong, all remotely and on your own schedule.

As part of Outlier, you will help shape how advanced AI systems reason, learn, and improve by providing evaluation feedback that feeds into model improvement.

Qualifications

• Fluent English with strong written communication and comfortable with everyday speech.
• Excellent attention to detail and the patience to replay or zoom in when something seems off.
• Able to think on your feet and discuss realistic scenarios naturally.
• Experience writing concise, evidence-based explanations or reviews.

Responsibilities

• Hold live, scenario-based conversations in English with AI voice models and rank naturalness, usefulness, and clarity.
• Compare pairs of AI-generated images/videos and judge which follows the prompt better.
• Inspect outputs closely to spot problems like distorted anatomy, flickering, text errors, or audio-video mismatch.
• Write specific explanations in English that point to what you saw or heard.

Skills

Fluent English

Attention to detail

Clear communication

Evidence-based writing

Decision making

Education

Bachelor’s degree or above

Job description

Outlier is an AI training platform owned and operated by Scale AI. We are seeking detail-oriented English speakers to review AI-generated media, roleplaying live conversations with an AI voice model and scoring each exchange, and explaining what looked or sounded wrong, all remotely and on your own schedule.

As part of Outlier, you will help shape how advanced AI systems reason, learn, and improve by providing evaluation feedback that feeds into model improvement.