Senior AI Voice Quality & Evaluation Lead (Remote)

Spotify Los Angeles; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Extensive learning opportunities

GreenHouse team

Flexible share incentives

Global parental leave

All The Feels employee assistance

Flexible public holidays

Job summary

Spotify in London is seeking a Senior Product Quality Analyst to lead how we define and measure AI voice quality across products, audiences, languages, and use cases. You will design evaluation rubrics and translate qualitative and quantitative findings into actionable recommendations for Voice Production, Speak and product teams.

You’ll establish ongoing evaluation rhythms, benchmark against third-party models, and help set quality trade-offs as we expand to more languages and markets.

Qualifications

• Experience in product quality, evaluation, data curation or a closely related field where human judgment assesses model or product performance.
• Ability to independently define evaluation criteria and turn subjective qualities into repeatable rubrics.
• Experience designing and running both qualitative and MOS-style evaluations and synthesizing results into actionable insights.

Responsibilities

• Own the quality bar for Spotify's AI voice models, defining what 'good enough' means for different products, audiences, languages and use cases.
• Design evaluation rubrics and an evaluation approach that measures naturalness, tone, pacing, pronunciation, accent, audio quality and overall experience.
• Plan and carry out evaluation campaigns, turning findings into recommendations for Voice Production, Speak and product teams.
• Establish an ongoing evaluation rhythm as technology evolves and products expand.
• Benchmark Spotify’s AI voices against third-party models and competitors to identify strengths and gaps.
• Help determine quality trade-offs as Spotify expands into additional languages and regions.
• Collaborate with voice experts, ML teams, product partners and vendors to maintain a consistent quality bar.

Skills

Product quality

Evaluation

Data curation

Qualitative evaluation

Job description

Spotify in London is seeking a Senior Product Quality Analyst to lead how we define and measure AI voice quality across products, audiences, languages, and use cases. You will design evaluation rubrics and translate qualitative and quantitative findings into actionable recommendations for Voice Production, Speak and product teams.

You’ll establish ongoing evaluation rhythms, benchmark against third-party models, and help set quality trade-offs as we expand to more languages and markets.