Voice AI Quality Evaluator

Innodata Inc.; Arkansas; On-site; USD 28,000 - 37,000; Full time

Job summary

Innodata Inc. in the United States seeks detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. You will compare two models using the same scenario and provide structured evaluations across five criteria.

You will record your voice, maintain balanced turns, identify error clusters, and write a detailed rationale for your final model preference, applying evaluation guidelines consistently to ensure a fair comparison.

Qualifications

• Bachelor's degree required.
• Strong listening and comprehension skills.
• Excellent written communication.
• Ability to follow detailed evaluation guidelines.
• Comfort speaking naturally and roleplaying different conversational scenarios.
• Ability to compare two interactions fairly without personal bias.
• Strong critical thinking and analytical skills.
• Reliability and consistency in structured evaluation tasks.
• Familiarity with AI assistants, voice-based AI, or conversational systems.

Responsibilities

• Review an assigned scenario and roleplay a natural conversation with two different AI models.
• Conduct comparable conversations with Model A and Model B, keeping scenario and approach consistent.
• Maintain approximately the same number of conversational turns with each model.
• Record your voice during each interaction and observe model responses.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize relevant error clusters in audio or conversation.
• Compare performance and select the stronger model based on criteria.
• Write a detailed rationale referencing observations from both conversations.
• Apply guidelines consistently across different scenarios.

Skills

Attention to detail

Listening skills

Written communication

Following evaluation guidelines

Roleplaying conversations

Fair/unbiased evaluation

Critical thinking

Reliability

Familiarity with AI assistants

Education

Bachelor's Degree

Job description

Innodata Inc. in the United States seeks detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. You will compare two models using the same scenario and provide structured evaluations across five criteria.

You will record your voice, maintain balanced turns, identify error clusters, and write a detailed rationale for your final model preference, applying evaluation guidelines consistently to ensure a fair comparison.