Remote Multilingual AI Quality & Evaluation Lead

Creandum Advisor LLP; Stockholms kommun; Hybrid; SEK 776,397 - 1,358,695; Full time

Job summary

Spotify is seeking a professional to advance multilingual AI data quality across Localization. You will define quality standards, design evaluations, and curate multilingual datasets to support scalable AI model development and evaluation.

Based in London or Stockholm, you will partner with Product, Engineering, Data Science, Research, and Localization to drive quality signals, use cases, and launch decisions, while embracing flexible work arrangements.

Qualifications

• Experience in multilingual quality evaluation, localization, data curation, annotation, or related fields.
• Understanding language quality, cultural relevance, and user experience across languages and markets.
• Familiar with structured evaluations using rubrics, audits, or annotation workflows.
• Comfort using qualitative and quantitative data to identify trends and improve quality.
• Knowledge of LLMs, AI evaluation, and human‑in‑the‑loop workflows is a plus.
• Ability to work in cross‑functional environments.

Responsibilities

• Define quality frameworks, rubrics, thresholds for multilingual AI experiences.
• Design and execute structured evaluations for AI-generated experiences.
• Lead multilingual dataset curation, annotation, and ground-truth creation.
• Analyze results, identify gaps, and provide actionable recommendations.
• Support evaluator calibration and human‑AI agreement studies.
• Collaborate with Product, Engineering, Data Science, Localization, vendors and markets.
• Document best practices and help define quality standards.
• Contribute to scalable evaluation capabilities for next-gen AI experiences.

Skills

Quality evaluation

Localization

Data curation

Annotation

AI evaluation

LLMs awareness

Tools

SQL

Python

Dashboards

Annotation platforms

Job description

Spotify is seeking a professional to advance multilingual AI data quality across Localization. You will define quality standards, design evaluations, and curate multilingual datasets to support scalable AI model development and evaluation.

Based in London or Stockholm, you will partner with Product, Engineering, Data Science, Research, and Localization to drive quality signals, use cases, and launch decisions, while embracing flexible work arrangements.