Remote AI Voice Evaluator - Model Quality Analyst

Innodata; United States; Remote; USD 28,000 - 37,000; Part time

Job summary

Innodata is seeking detail-oriented Voice Specialists to evaluate AI models through real-time voice conversations. You will interact with two models using the same scenario, compare responses, and provide structured evaluations based on defined quality criteria.

The role involves recording your voice during each interaction, identifying error clusters, and writing a detailed rationale of your final preference.

Qualifications

• Detail-oriented with ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Strong written communication to explain observations clearly and objectively.
• Ability to follow detailed evaluation guidelines consistently.
• Comfort speaking naturally and roleplaying different conversational scenarios.
• Familiarity with AI assistants or voice-based systems.

Responsibilities

• Review an assigned scenario and roleplay a natural conversation with two different AI models.
• Conduct comparable conversations with Model A and Model B, ensuring consistency in scenario and approach.
• Maintain approximately the same number of turns for a fair comparison.
• Record your voice during each interaction and assess model responses.
• Identify and categorize errors that may occur in audio or conversation.
• Write a detailed rationale explaining final preference with concrete examples from both conversations.
• Apply evaluation guidelines consistently across different scenarios and model interactions.

Skills

Attention to detail

Listening skills

Written communication

Analytical thinking

Consistency

Familiarity with voice AI

Job description

Innodata is seeking detail-oriented Voice Specialists to evaluate AI models through real-time voice conversations. You will interact with two models using the same scenario, compare responses, and provide structured evaluations based on defined quality criteria.

The role involves recording your voice during each interaction, identifying error clusters, and writing a detailed rationale of your final preference.