Real-Time Voice AI Evaluator — Contract

OpenKrill; United States; Remote; USD 42,000 - 52,000; Full time

Job summary

Innodata is seeking a detail-oriented Voice Specialist for a 3-month, W2, full-time contract in the United States. You will evaluate two AI models through real-time conversations, comparing responses and providing structured feedback across defined criteria.

Responsibilities include role-playing with Model A and Model B, maintaining balanced turns, recording and analyzing voice interactions, and delivering a detailed rationale with objective observations.

Qualifications

• Bachelor's Degree required and a strong eye for detail.
• Excellent listening and comprehension abilities are essential.
• Clear written communication to explain observations objectively.
• Ability to follow detailed evaluation guidelines reliably.
• Comfort with role-playing conversations and evaluating two models fairly.
• Familiarity with AI assistants, voice-based AI, or conversational systems.

Responsibilities

• Review an assigned scenario and roleplay conversations with two AI models.
• Conduct comparable conversations with Model A and Model B with consistent approach.
• Maintain similar conversational turns for fairness across models.
• Record your voice during interactions and assess model behavior.
• Evaluate models across defined dimensions and categorize error clusters.
• Compare model performances and select the stronger overall option.
• Write a detailed rationale referencing observed examples.
• Apply evaluation guidelines consistently across scenarios.

Skills

Attention to detail

Listening skills

Written communication

Critical thinking

Analytical skills

Education

Bachelor's degree

Job description

Innodata is seeking a detail-oriented Voice Specialist for a 3-month, W2, full-time contract in the United States. You will evaluate two AI models through real-time conversations, comparing responses and providing structured feedback across defined criteria.

Responsibilities include role-playing with Model A and Model B, maintaining balanced turns, recording and analyzing voice interactions, and delivering a detailed rationale with objective observations.