Conversational AI Voice Evaluation Analyst

Innodata; United States; Remote; USD 28,000 - 34,000; Full time

Job summary

Innodata is seeking a detail-oriented Voice Specialist for a 3-month, full-time W2 contract (40 hours per week) in the United States. You will evaluate AI model conversations in real-time, comparing two models, and provide structured evaluations across five dimensions.

You'll record your responses, maintain consistency across conversations, identify error clusters, and write a detailed rationale favoring one model over the other based on observed performance.

Qualifications

• Bachelor's Degree required.
• Strong attention to detail and ability to notice subtle differences.
• Excellent listening and comprehension skills.
• Strong written communication, especially the ability to explain observations clearly and objectively.
• Ability to follow detailed evaluation guidelines.
• Comfort speaking naturally and role-playing different conversational scenarios.
• Ability to compare two interactions fairly without allowing personal preferences to influence the evaluation.
• Familiarity with AI assistants, voice-based AI, or conversational systems.

Responsibilities

• Review scenarios and roleplay conversations with two AI models.
• Conduct comparable conversations with Model A and Model B.
• Maintain approximately the same number of turns for fairness.
• Record your voice during interactions and observe model responses.
• Evaluate models across five defined evaluation dimensions.
• Identify and categorize error clusters in audio/conversation.
• Compare performance and select the stronger model.
• Provide a detailed rationale with observations from both conversations.

Skills

Attention to detail

Listening skills

Written communication

Following evaluation guidelines

Critical thinking

Analytical skills

Reliability

AI/voice familiarity

Education

Bachelor's Degree

Job description

Innodata is seeking a detail-oriented Voice Specialist for a 3-month, full-time W2 contract (40 hours per week) in the United States. You will evaluate AI model conversations in real-time, comparing two models, and provide structured evaluations across five dimensions.

You'll record your responses, maintain consistency across conversations, identify error clusters, and write a detailed rationale favoring one model over the other based on observed performance.