Multilingual AI Quality Lead — Global Platform

Spotify; Stockholms kommun; Hybrid; SEK 850,000 - 1,150,000; Full time

Job summary

Spotify is seeking a dedicated professional to join the Multilingual AI Data Quality space within Localization. You will help define quality standards for multilingual AI experiences, design structured evaluations, and lead dataset curation and ground-truth creation to support AI model development.

You will collaborate with product, engineering, data science, research, localization, and external partners to improve quality signals and inform launch decisions, while contributing to scalable

Qualifications

• Experience in multilingual quality evaluation, localization, data curation, annotation, or AI evaluation.
• Understanding language quality, cultural relevance, content quality, and user experience across languages and markets.
• Experience designing or conducting structured evaluations using quality rubrics, audits, annotation projects, or review methodologies.
• Able to use qualitative and quantitative data to identify trends, measure quality, and make recommendations.
• Familiar with large language models (LLMs), generative AI evaluation, human-in-the-loop workflows, or LLM-as-a-judge methodologies.
• Comfort with ambiguity and turning complex quality challenges into practical evaluation strategies.
• Communication skills and ability to collaborate across technical and non-technical partners.
• Experience with recommendation systems, personalization, search, ranking, machine translation, generative AI, dataset creation, annotation operations, evaluator calibration, prompt testing, model evaluation, SQL, Python, dashboards, or annotation platforms is a plus.

Responsibilities

• Define quality frameworks, evaluation rubrics, thresholds, and methodologies for multilingual AI experiences.
• Design and execute structured evaluations for AI-generated, AI-translated, AI-curated, and recommendation-driven experiences.
• Lead multilingual dataset curation, annotation, enrichment, and ground-truth creation to support AI model development and evaluation.
• Analyze evaluation results, identify quality gaps, and provide actionable recommendations to improve multilingual AI quality.
• Support LLM-as-a-judge workflows, evaluator calibration, and human-AI agreement studies.
• Partner closely with Product, Engineering, Data Science, Research, Localization, vendors, and market experts to improve AI quality signals and inform launch decisions.
• Document best practices and help define quality standards across languages, markets, and AI use cases.
• Contribute to building scalable evaluation capabilities that support the next generation of AI-powered experiences across Spotify.

Skills

Multilingual quality evaluation

Localization

Data curation

Annotation

AI evaluation

Structured evaluations

Qualitative analysis

Quantitative analysis

Cross-functional collaboration

LLMs

SQL

Python

Tools

SQL

Python

Annotation platforms

Job description

Spotify is seeking a dedicated professional to join the Multilingual AI Data Quality space within Localization. You will help define quality standards for multilingual AI experiences, design structured evaluations, and lead dataset curation and ground-truth creation to support AI model development.

You will collaborate with product, engineering, data science, research, localization, and external partners to improve quality signals and inform launch decisions, while contributing to scalable