Welsh Audio Evaluation Specialist

Agency; United States; Remote; USD 60,000 - 90,000; Full time

Job summary

Agency seeks an evaluator to design and run advanced role-play scenarios that mimic real customer service interactions. You will assess AI models on task completion, natural language quality, and basic JSON reasoning.

The role requires Welsh proficiency and prior experience in complex support or AI evaluation, with access to reliable audio input equipment. You will help build diverse audio datasets for real-world scenarios.

Qualifications

• Native or bilingual proficiency in Welsh.
• Experience in complex customer support or AI evaluation.
• Basic JSON and programming literacy.
• Access to a high-quality microphone for clear audio input.
• Meticulous, detail-oriented approach to structured prompts and rubrics.

Responsibilities

• Create and execute complex, role-play-based evaluation scenarios simulating realistic customer service interactions.
• Evaluate AI model performance across qualitative and quantitative metrics, focusing on task completion and conversational naturalness.
• Assess model's basic programming literacy, including JSON structures and reasoning about structured data.
• Contribute to the development of diverse, high-quality audio datasets reflecting real customer expectations.

Skills

Customer support

AI evaluation

Data analysis

Attention to detail

Tools

JSON literacy

Audio tools

Job description

Agency seeks an evaluator to design and run advanced role-play scenarios that mimic real customer service interactions. You will assess AI models on task completion, natural language quality, and basic JSON reasoning.

The role requires Welsh proficiency and prior experience in complex support or AI evaluation, with access to reliable audio input equipment. You will help build diverse audio datasets for real-world scenarios.