Evaluation Science Lead

Apple Inc.; Austin (TX); On-site; USD 170,000 - 210,000; Full time

Job summary

Apple Inc. in Austin, Texas seeks an Evaluation Science Lead to define how multilingual Globalization AI solutions are measured and validated across Apple Services, including Apple Music and App Store.

This is a rare opportunity to build evaluation science at the intersection of language, culture, and AI. You will design statistically grounded evaluation frameworks combining human judgment with scalable automation, partnering with Engineering, AI Strategy, and Production to turn evaluation into

Qualifications

• 5+ years of experience in evaluation science, data science, or ML systems development with end-to-end ownership of evaluation systems.
• Experience applying statistical methodologies (sampling, significance testing, confidence intervals) and measurement validity principles.
• Hands-on experience measuring annotator agreement and improving annotation protocols.

Responsibilities

• Define the long-term evaluation science strategy and roadmap for Globalization AI solutions across 50+ languages.
• Design robust statistical methodologies and scalable evaluation workflows with human annotation and automated scoring.
• Translate evaluation data into actionable recommendations and go/no-go evidence for Engineering and leadership.
• Build continuous performance monitoring and drift-detection to distinguish real issues from noise.
• Serve as the resident authority on multilingual AI evaluation and promote best practices.

Skills

Evaluation science

Data science

ML systems development

Education

Master’s or PhD or equivalent

Tools

R

Python

SQL

Job description

Austin, Texas, United States Machine Learning and AI

The people here at Apple don’t just create products — they create the kind of wonder that’s revolutionized entire industries. It’s the diversity of those people and their ideas that inspires the innovation that runs through everything we do. Join Apple, and help us leave the world better than we found it.We are looking for an Evaluation Science Lead to define how our multilingual Globalization AI solutions are measured and validated across Apple Services — including Apple Music, App Store, Apple TV+, Apple Podcasts, and more. This is a rare opportunity to build evaluation science as a discipline at the intersection of language, culture, and AI, and to directly shape whether hundreds of millions of users around the world feel genuinely at home in Apple's products.

Description

Language carries culture, nuance, and context — as Evaluation Science Lead, you will own the scientific rigor behind measuring how our AI-powered Globalization solutions perform across 50+ languages and dozens of markets. As a strategic individual contributor on the Globalization Quality and Operations team, you will design statistically grounded evaluation frameworks combining human judgment with scalable automation — partnering with Engineering, AI Strategy, and Production to turn evaluation into a strategic capability.This role requires strong analytical and scientific thinking, hands-on execution, and sound judgment to communicate complex findings clearly. You thrive in ambiguity and know evaluation delivers value only when operationalized — if you believe rigorous, culturally-informed evaluation is one of the highest-leverage ways to improve AI language quality globally, this role was built for you.

Responsibilities

• Define the long-term evaluation science strategy and roadmap for Globalization AI solutions across 50+ languages, including targeted approaches for low-resource locales.
• Design robust statistical methodologies—including power analysis, sampling, confidence intervals, and significance thresholds—to evaluate models with scientific validity.
• Architect scalable evaluation workflows blending human annotation (calibration, protocols, inter-annotator agreement) with automated systems (autograders, LLM-as-judge, and agent-based first-pass scoring).
• Expand evaluation criteria beyond core linguistic quality to incorporate behavioral and user engagement signals.
• Insights & Continuous Improvement
• Translate complex evaluation data and loss patterns into clear, actionable recommendations and go/no-go evidence for Engineering and leadership.
• Build continuous performance monitoring and drift-detection systems to distinguish real regressions from measurement noise.
• Establish feedback loops with Quality Operations and Engineering to rapidly refine evaluation rubrics as models evolve.
• Serve as the resident authority on multilingual AI evaluation, communicating methodologies and upleveling best practices across Globalization Apple Services.

Minimum Qualifications

• 5+ years of experience in evaluation science, data science, or ML systems development, with demonstrated experience owning evaluation systems at scale end-to-end
• Experience applying statistical methodology — sampling, significance testing, and confidence intervals
• Practical understanding of measurement validity principles
• Hands-on experience measuring annotator agreement, diagnosing divergence, and improving annotation protocols
• Proficiency with statistical tools and languages (R, Python, SQL) for analysis and reproducibility
• Experience designing evaluation frameworks adopted and scaled by operational teams
• Strong written and verbal communication skills for technical and non-technical audiences
• Direct experience handling sensitive and confidential information with integrity and discretion
• Ability to be onsite; this role is an in-person, onsite position
• Availability to work occasional evenings and weekends, as business needs require
• Up to 10% + travel; both domestic and international

Preferred Qualifications

• Master’s, PhD, or comparable experience in Statistics, Computational Linguistics, Computer Science, Psychometrics, Data Science, or related quantitative field.
• Experience evaluating generative AI systems—including hallucination detection, safety and cultural alignment, autograders / LLM-as-judge systems, benchmark design, and synthetic data evaluation.
• Experience with A/B testing, causal inference, or experimental design
• Experience in applied linguistics — translating cultural and linguistic nuances into quantitative evaluation framework

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong. Learn about accessibility in Apple’s workplace Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.