Voice AI Benchmarking Analyst

Socket.dev; Alabama; On-site; USD 28,000 - 37,000; Full time

Job summary

Innodata is seeking a detail-oriented Voice Specialist to evaluate AI model performance through real-time voice conversations. You will interact with two models using the same scenario, compare responses, and provide structured evaluations based on five defined quality criteria.

You will record your voice during each interaction, identify error clusters, and write a clear rationale selecting the stronger model.

Qualifications

• Bachelor's Degree required.
• Excellent listening and comprehension skills.
• Strong written communication to explain observations clearly.
• Ability to follow detailed evaluation guidelines consistently.
• Comfort speaking naturally and roleplaying different conversational scenarios.
• Reliability and consistency during structured evaluation tasks.

Responsibilities

• Review an assigned scenario and roleplay a natural conversation with two AI models.
• Conduct comparable conversations with Model A and Model B, keeping the scenario consistent.
• Maintain approximately the same number of conversational turns for a fair comparison.
• Record your voice during each interaction and evaluate model responses.
• Evaluate each model across five defined evaluation dimensions.
• Identify and categorize relevant error clusters in audio or conversation.
• Compare performance and select the stronger model based on evaluation criteria.
• Write a detailed rationale referencing examples from both conversations.
• Apply guidelines consistently across different scenarios and model interactions.

Skills

Attention to detail

Listening skills

Written communication

Follow evaluation guidelines

Roleplaying conversations

Critical thinking

Analytical skills

Reliability

Familiarity with voice-based AI

Education

Bachelor's Degree

Job description

Innodata is seeking a detail-oriented Voice Specialist to evaluate AI model performance through real-time voice conversations. You will interact with two models using the same scenario, compare responses, and provide structured evaluations based on five defined quality criteria.

You will record your voice during each interaction, identify error clusters, and write a clear rationale selecting the stronger model.