Voice AI Quality Evaluator for Conversational Systems

Innodata Inc.; Maryland City (MD); On-site; USD 28,000 - 34,000; Full time

Job summary

Innodata Inc. seeks detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. You will compare two models on predefined criteria and document your analyses and conclusions.

Responsibilities include roleplaying conversations, recording sessions, and identifying error clusters to inform which model delivers a stronger overall conversational experience. Bachelor’s degree required.

Qualifications

• Bachelors Degree is required.
• Strong attention to detail and ability to notice subtle differences in conversational quality.
• Excellent listening and comprehension skills.
• Strong written communication skills to explain observations clearly and objectively.
• Ability to follow detailed evaluation guidelines consistently.
• Comfort speaking naturally and roleplaying different conversational scenarios.
• Ability to compare two interactions fairly without personal bias.
• Strong critical-thinking and analytical skills.
• Reliability and consistency when completing structured evaluation tasks.
• Familiarity with AI assistants, voice-based AI, or conversational systems.

Responsibilities

• Review and roleplay conversations with two AI models using the same scenario.
• Conduct comparable conversations with Model A and Model B ensuring consistency.
• Maintain approximately the same number of turns for fairness.
• Record your voice during each interaction and observe model responses.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize error clusters that may occur in audio or conversation.
• Compare performance and select the model with stronger overall performance.
• Write a detailed rationale referencing specific observations from both conversations.
• Apply evaluation guidelines consistently across scenarios.

Skills

Detail-oriented

Excellent listening

Comprehension

Written communication

Follow evaluation guidelines

Roleplay conversations

Critical thinking

Analytical skills

Reliability

Familiar with voice AI

Education

Bachelor's Degree

Job description

Innodata Inc. seeks detail-oriented Voice Specialists to evaluate AI model performance through real-time, voice-based conversations. You will compare two models on predefined criteria and document your analyses and conclusions.

Responsibilities include roleplaying conversations, recording sessions, and identifying error clusters to inform which model delivers a stronger overall conversational experience. Bachelor’s degree required.