Voice AI Quality Evaluator

Innodata; United States; Remote; USD 28,000 - 34,000; Full time

Job summary

Innodata seeks detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. You will interact with two AI models using the same scenario, compare responses, and provide structured evaluations.

You will record your voice, assess quality across defined dimensions, identify error clusters, and write a final rationale selecting the stronger model. Bachelor’s degree required; strong listening and written communication skills are essential.

Qualifications

• Bachelor’s degree required.
• Strong attention to detail and ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.

Responsibilities

• Review an assigned scenario and roleplay conversations with two AI models.
• Conduct comparable conversations with Model A and Model B, keeping the scenario and approach consistent.
• Maintain approximately the same number of turns with each model for a fair comparison.
• Record your voice during interactions and evaluate the model’s responses.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize relevant error clusters from audio or conversation.
• Compare model performance and select the stronger overall.
• Write a detailed rationale referencing observations from both conversations.
• Apply evaluation guidelines consistently across scenarios.

Skills

Attention to detail

Listening skills

Written communication

Evaluation guidelines

Critical thinking

Reliability

AI assistants

Education

Bachelor’s degree

Job description

Innodata seeks detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. You will interact with two AI models using the same scenario, compare responses, and provide structured evaluations.

You will record your voice, assess quality across defined dimensions, identify error clusters, and write a final rationale selecting the stronger model. Bachelor’s degree required; strong listening and written communication skills are essential.