AI Voice Quality Evaluator

Innodata Inc.; New York (NY); On-site; USD 28,000 - 37,000; Full time

Job summary

Innodata Inc. is seeking detail-oriented Voice Specialists to evaluate AI models via real-time voice conversations. You will compare two models, record your interactions, and provide structured evaluations across five criteria to determine the stronger overall conversational experience.

The role emphasizes maintaining consistent turns, careful listening, and objective reporting, with the goal of delivering fair, actionable insights for model improvement.

Qualifications

• Bachelor's degree required.
• Strong attention to detail and ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Clear, objective written observations and reporting.
• Ability to follow detailed evaluation guidelines and document results.

Responsibilities

• Review an assigned scenario and roleplay with two different AI models.
• Conduct comparable conversations with Model A and Model B with consistent approach.
• Maintain similar number of turns for a fair comparison.
• Record voice during each interaction and assess model responses.
• Evaluate across five defined evaluation dimensions.
• Identify and categorize error clusters in audio or conversation.
• Compare performance and select the stronger model.
• Write a detailed rationale referencing observations from both conversations.
• Apply guidelines consistently across scenarios.

Skills

Detail-oriented

Excellent listening

Strong written communication

Following evaluation guidelines

Critical thinking

Education

Bachelor's Degree

Tools

AI assistants

Job description

Innodata Inc. is seeking detail-oriented Voice Specialists to evaluate AI models via real-time voice conversations. You will compare two models, record your interactions, and provide structured evaluations across five criteria to determine the stronger overall conversational experience.

The role emphasizes maintaining consistent turns, careful listening, and objective reporting, with the goal of delivering fair, actionable insights for model improvement.