Remote Audio AI Evaluator - Customer Support Benchmarks

Agency; United States; Remote; USD 15,000 - 41,000; Full time

Job summary

Agency manages a program to assess and benchmark advanced agentic audio models for real-world customer support. The role focuses on creating realistic evaluation scenarios across travel, finance, telecom, and tech support to test conversational audio agents.

Responsibilities include building diverse datasets and applying structured evaluation metrics to measure performance like clarity, efficiency, and naturalness of responses, ensuring alignment with customer expectations across domains.

Qualifications

• Understanding of JSON structures.
• Familiarity with functions and methods.
• Ability to reason about structured data and simple logic.

Responsibilities

• Create and execute role-play–based evaluation scenarios across domains such as flight bookings, travel support, financial services, telecommunications, and technical support.
• Contribute to development of diverse and representative datasets for evaluating conversational audio agents.
• Evaluate model performance using a standardized set of qualitative and quantitative metrics.
• Ensure evaluations reflect real customer expectations for clarity, efficiency, and natural conversational flow.

Skills

JSON structures understanding

Functions and methods familiarity

Structured data reasoning

Job description

Agency manages a program to assess and benchmark advanced agentic audio models for real-world customer support. The role focuses on creating realistic evaluation scenarios across travel, finance, telecom, and tech support to test conversational audio agents.

Responsibilities include building diverse datasets and applying structured evaluation metrics to measure performance like clarity, efficiency, and naturalness of responses, ensuring alignment with customer expectations across domains.