Voice AI Quality Evaluator

Innodata Inc.; Massachusetts; On-site; USD 28,000 - 37,000; Full time

Job summary

Innodata Inc. is seeking detail-oriented Voice Specialists to evaluate AI models through real-time, voice-based conversations. You will interact with two AI models, compare their performance, and provide structured evaluations based on five defined criteria.

Responsibilities include roleplaying, maintaining balanced turns, recording audio, and identifying error clusters to determine which model delivers a stronger conversational experience. Bachelor’s degree required.

Qualifications

• Bachelor's Degree required.
• Excellent listening and comprehension skills.
• Strong written communication abilities.
• Ability to follow detailed evaluation guidelines.
• Familiarity with AI assistants or conversational systems is a plus.

Responsibilities

• Review assigned scenarios and roleplay a natural conversation with two AI models.
• Conduct comparable conversations with Model A and Model B, keeping scenario and approach consistent.
• Maintain approximately the same number of conversational turns with each model.
• Record voice during interactions and evaluate model responses across defined criteria.
• Identify and categorize relevant error clusters observed in audio or conversation.
• Provide a detailed rationale for final model preference with concrete observations.
• Apply evaluation guidelines consistently across different scenarios.

Skills

Attention to detail

Listening comprehension

Written communication

Fair evaluation & objectivity

Critical thinking

Education

Bachelor's Degree

Job description

Innodata Inc. is seeking detail-oriented Voice Specialists to evaluate AI models through real-time, voice-based conversations. You will interact with two AI models, compare their performance, and provide structured evaluations based on five defined criteria.

Responsibilities include roleplaying, maintaining balanced turns, recording audio, and identifying error clusters to determine which model delivers a stronger conversational experience. Bachelor’s degree required.