Italian Audio Evaluations Specialist

Agency; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Agency is seeking individuals to create and run advanced evaluation scenarios for customer service interactions. You will assess AI model performance across qualitative and quantitative metrics, focusing on task completion and naturalness, while evaluating literacy in JSON structures.

The role includes contributing to diverse, high-quality audio data collection. Ideal candidates have native Italian proficiency, a background in customer support or AI evaluation, and basic programming literacy.

Qualifications

• Native or bilingual proficiency in Italian.
• Experience in customer support, technical troubleshooting, or conversational AI evaluation.
• Basic programming literacy with JSON, functions, methods, and simple logic.
• Access to a high-quality microphone.
• Meticulous attention to detail with structured prompts and evaluation rubrics.

Responsibilities

• Create and execute complex evaluation scenarios simulating customer service interactions.
• Evaluate AI model performance using qualitative and quantitative metrics.
• Assess model programming literacy, including JSON structures and reasoning about structured data.
• Contribute to the development of diverse, high-quality audio datasets.

Skills

Native Italian

AI evaluation experience

Customer service background

JSON basics

Attention to detail

Tools

JSON

Python basics

Job description

Responsibilities

• Create and execute complex, role-play-based evaluation scenarios simulating realistic customer service interactions
• Evaluate AI model performance across qualitative and quantitative metrics, focusing on task completion and naturalness
• Assess model programming literacy, including JSON structures and reasoning about structured data
• Contribute to the development of diverse, high-quality audio datasets

Requirements

• Native or bilingual proficiency in Italian
• Professional expertise in customer support, technical troubleshooting, or conversational AI evaluation
• Basic computer programming literacy (JSON, functions, methods, and simple logic)
• Access to a high-quality microphone for clean audio input
• Meticulous attention to detail with structured prompts and evaluation rubrics

About the Company

This project focuses on assessing advanced agentic audio models to optimize performance in real-world customer support scenarios like flight bookings, financial services, and telecommunications.