Dutch Audio AI Evaluator for Customer Support Metrics

Agency; United States; Remote; USD 60,000 - 90,000; Full time

Job summary

Agency is seeking a Dutch-speaking evaluator to design and execute role-played customer service simulations and judge AI model performance across qualitative and quantitative metrics. The role emphasizes task completion, conversational naturalness, and basic programming literacy.

You will contribute to audio dataset development that reflects real customer expectations and support high-quality audio input during evaluations, with potential remote data collection elements.

Qualifications

• Native or bilingual proficiency in Dutch.
• Professional expertise in complex customer support, technical troubleshooting, or conversational AI evaluation.
• Basic computer programming literacy (JSON, functions, methods, and simple logic).
• Access to a high-quality microphone for clean audio input.
• Meticulous, detail-oriented approach to structured prompts and evaluation rubrics.

Responsibilities

• Create and execute complex, role-play-based evaluation scenarios simulating realistic customer service interactions.
• Evaluate AI model performance across qualitative and quantitative metrics, focusing on task completion and conversational naturalness.
• Assess model's basic computer programming literacy, including JSON structures and reasoning about structured data.
• Contribute to the development of diverse, high-quality audio datasets reflecting real customer expectations.

Skills

Dutch language

Customer support / AI evaluation

Programming basics (JSON)

Audio data collection

Tools

JSON

Programming basics

Job description

Agency is seeking a Dutch-speaking evaluator to design and execute role-played customer service simulations and judge AI model performance across qualitative and quantitative metrics. The role emphasizes task completion, conversational naturalness, and basic programming literacy.

You will contribute to audio dataset development that reflects real customer expectations and support high-quality audio input during evaluations, with potential remote data collection elements.