AI Measurement Scientist, AI Evaluation Platform

Apple, Inc.

Seattle (WA)

Sur place

USD 142 000 - 263 000

Plein temps

Il y a 16 heures
Soyez parmi les premiers à postuler

Une candidature complète en une minute

Un CV et une lettre de motivation personnalisés, prêts à envoyer.

Passez les filtres ATS

Résumé du poste

Apple in Seattle is seeking an AI Measurement Scientist to build the scientific foundation for evaluating AI systems across Apple products and services. You will apply state-of-the-art measurement methods, experimental design, and causal inference to establish valid, trustworthy evaluation standards and to produce actionable signals for engineering teams.

You will work with ML researchers and platform engineers to productionize measurement tools, create self-service validity checks, and publish

Qualifications

  • PhD in Computer Science or Machine Learning with a measurement or evaluation focus, Psychometrics, Educational Measurement, Statistics, Biostatistics, Econometrics, or equivalent research/work experience.
  • Deep expertise in modeling test or benchmark data (IRT, latent variable, or other response models) and construct validation, in benchmark design and validity for AI systems, or in experimental design and causal inference (surrogate outcomes, observational causal methods).
  • Strong statistical foundation including experimental design, power analysis, sampling theory, and uncertainty quantification.
  • Track record of designing and validating measurement instruments or experiments as demonstrated through publications or applied work.
  • Proficiency in Python (preferred) or R for statistical analysis, psychometric modeling, and method implementation.
  • Strong working knowledge of generative AI technology.
  • Excellent communication skills with the ability to explain complex measurement concepts to engineers, ML researchers, and non-technical stakeholders.

Responsabilités

  • Design validity frameworks for AI evaluation, ensuring that automated metrics, LLM-as-judge systems, and human evaluation protocols measure what they claim to measure across diverse contexts.
  • Develop and apply methods from measurement science to assessing the quality of benchmarks, for example drawing on frameworks like item response theory (IRT).
  • Create calibration and bias detection systems for automated evaluators, ensuring LLM-as-judge scores are interpretable, consistent, and free from systematic biases.
  • Build robust statistical tools for practitioners for sample-size planning, comparing system versions, quantifying uncertainty, controlling error rates, and visualizing data.
  • Connect offline evaluation to real-world outcomes: test whether offline score changes predict what users experience, design randomized audits, and develop methods for setting defensible ship thresholds that engineering teams can apply.
  • Establish measurement standards for evaluator transfer and generalization, including methods to quantify or predict when evaluators will maintain validity across domains, languages, or contexts.
  • Validate novel evaluation methods in collaboration with ML researchers, ensuring intelligent search algorithms discover statistically meaningful patterns, failure analyses estimate how often problems occur with uncertainty, and synthetic data generation produces representative samples.
  • Collaborate with platform engineers to productionize measurement methods into evaluation infrastructure, creating self-service tools for validity checking, reliability testing, and interpretable outputs (report cards, warnings, confidence metrics).
  • Publish research at top measurement venues and/or ML conferences (NeurIPS, ICML, ICLR), advancing both measurement science and AI evaluation.
  • Collaborate across disciplines with ML researchers developing novel methods, platform engineers building scalable infrastructure, and evaluation practitioners using these tools in production.

Connaissances

Python (preferred)
Statistics
Experimental design
Communication skills

Formation

PhD in Computer Science or Machine Learning

Outils

R
Jupyter

Description du poste

AI Measurement Scientist, AI Evaluation Platform

Seattle, Washington, United States Software and Services

Apple is where individual imaginations gather together, committing to the values that lead to great work. Every new product we build, service we create, or experience we deliver is the result of us making each other's ideas stronger.

Description

Our team, part of Apple Services Engineering, is building the scientific foundation for how AI systems are evaluated across Apple. Our goal is to create the evidence that teams can act on: what to measure, whether a change helped, why a system failed, and whether it is ready to ship. We are seeking a Measurement Scientist to ensure that our evaluation methods are not just sophisticated, but scientifically valid and trustworthy. In this role, you will apply apply state-of-the-art measurement methods, validity frameworks, experimental design, and statistical rigor to establish measurement standards for AI evaluation, ensuring that when we claim an evaluator measures "helpfulness" or "safety," it actually does. We are looking for individuals across a range of experience levels.This role uniquely bridges measurement science and cutting-edge AI evaluation. This role uniquely bridges measurement science and cutting-edge AI evaluation. Our team is building AI tools and capabilities to help developers do evaluation better and faster. You will develop methods for validating these tools: making sure that they generate evaluation artifacts and workflows that give developers the right signals to make sound engineering decisions and build great products for Apple customers.You will work on an interdisciplinary team with with other measurement scientists and ML researchers to solve new problems in AI evaluation. Your work will be both published at top measurement and ML venues and productionized into the evaluation SDK used across Apple. The successful candidate will have deep expertise in psychometrics or other measurement disciplines, or in experimental design and causal inference, with the ability to apply these principles to novel AI evaluation challenges. Our team deliberately combines ML science and measurement science. We especially welcome people who bridge those areas, including computer scientists who specialize in measurement, and measurement scientists who have worked on ML problems. You will also work collaboratively with platform engineers and evaluation practitioners to translate measurement science into practical tools that scale across the organization.

Responsibilities
  • Design validity frameworks for AI evaluation, ensuring that automated metrics, LLM-as-judge systems, and human evaluation protocols measure what they claim to measure across diverse contexts.
  • Develop and apply methods from measurement science to assessing the quality of benchmarks, for example drawing on frameworks like item response theory (IRT).
  • Create calibration and bias detection systems for automated evaluators, ensuring LLM-as-judge scores are interpretable, consistent, and free from systematic biases.
  • Build robust statistical tools for practitioners for sample-size planning, comparing system versions, quantifying uncertainty, controlling error rates, and visualizing data.
  • Connect offline evaluation to real-world outcomes: test whether offline score changes predict what users experience, design randomized audits, and develop methods for setting defensible ship thresholds that engineering teams can apply.
  • Establish measurement standards for evaluator transfer and generalization, including methods to quantify or predict when evaluators will maintain validity across domains, languages, or contexts.
  • Validate novel evaluation methods in collaboration with ML researchers, ensuring intelligent search algorithms discover statistically meaningful patterns, failure analyses estimate how often problems occur with uncertainty, and synthetic data generation produces representative samples.
  • Collaborate with platform engineers to productionize measurement methods into evaluation infrastructure, creating self-service tools for validity checking, reliability testing, and interpretable outputs (report cards, warnings, confidence metrics).
  • Publish research at top measurement venues and/or ML conferences (NeurIPS, ICML, ICLR), advancing both measurement science and AI evaluation.
  • Collaborate across disciplines with ML researchers developing novel methods, platform engineers building scalable infrastructure, and evaluation practitioners using these tools in production.
Minimum Qualifications
  • PhD in Computer Science or Machine Learning with a measurement or evaluation focus, Psychometrics, Educational Measurement, Quantitative Psychology, Statistics, Biostatistics, Econometrics, or equivalent research/work experience.
  • Deep expertise in modeling test or benchmark data (IRT, latent variable, or other response models) and construct validation, in benchmark design and validity for AI systems, or in experimental design and causal inference (surrogate outcomes, observational causal methods).
  • Strong statistical foundation including experimental design, power analysis, sampling theory, and uncertainty quantification.
  • Track record of designing and validating measurement instruments or experiments as demonstrated through publications or applied work.
  • Proficiency in Python (preferred) or R for statistical analysis, psychometric modeling, and method implementation.
  • Strong working knowledge of generative AI technology.
  • Excellent communication skills with the ability to explain complex measurement concepts to engineers, ML researchers, and non-technical stakeholders.
Preferred Qualifications
  • Research or experience applying measurement science to AI/ML evaluation, benchmark validity, automated scoring systems, or computational assessment.
  • Knowledge of modern ML evaluation challenges including LLM-as-judge, automated metrics, benchmark design, and agentic systems.
  • Publications at measurement venues or top ML conferences (NeurIPS, ICML, ICLR).
  • Expertise in computational social or behavioral science using generative AI.
  • Experience collaborating with engineers to turn research methods into production tools and scalable infrastructure.
  • Experience with online experimentation or with validating offline metrics against real-world outcomes (surrogate or proxy metrics).
  • Experience estimating rare events or making decisions when one kind of error costs far more than another.

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $142,300 and $263,300, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs.

Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan.

You’ll also receive benefits including:

  • Comprehensive medical and dental coverage
  • retirement benefits
  • a range of discounted products and free services
  • for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition

Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.

Learn about accessibility in Apple’s workplace

Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.

Obtenez votre examen gratuit et confidentiel de votre CV.

ou faites glisser et déposez votre fichier ici.

Similar jobs

Postes similaires à comparer

AI Applied Scientist, AI Evaluation Science
AI Applied Scientist, AI Evaluation Science

Apple, Inc. • Seattle (WA)

Sur place
USD 142 000 - 263 000
Machine Learning Engineer, Human Centered AI - Evaluations & Insights
Machine Learning Engineer, Human Centered AI - Evaluations & Insights

Apple Inc. • Seattle (WA)

Sur place
USD 142 000 - 263 000
AIML - Sr Data Scientist, Evaluation
AIML - Sr Data Scientist, Evaluation

Apple, Inc. • Cupertino (CA)

Sur place
USD 185 000 - 325 000
Medical and dental coverage
Stock program eligibility
Tuition reimbursement
+3
AIML - Sr Software Engineer - AI, Evaluation
AIML - Sr Software Engineer - AI, Evaluation

Apple, Inc. • Cupertino (CA)

Sur place
USD 150 000 - 278 000
Comprehensive medical and dental
Retirement benefits
Employee stock programs
+2
Sr. Applied Scientist, AI Evaluation & Quality Systems
Sr. Applied Scientist, AI Evaluation & Quality Systems

Apple Inc. • Seattle (WA)

Sur place
USD 142 000 - 263 000
Apple Benefits
Relocation assistance
Employee stock purchase plan
+2
Machine Learning Evaluation Engineer
Machine Learning Evaluation Engineer

Apple Inc. • Sunnyvale (CA)

Sur place
USD 150 000 - 278 000
Medical & Dental coverage
Employee stock programs
Relocation support
+1
Software Data Engineer, Sensing & Connectivity
Software Data Engineer, Sensing & Connectivity

Apple • Cupertino (CA)

Sur place
USD 150 000 - 278 000
Medical and dental coverage
Employee stock purchase plan
Relocation assistance
Research Engineer / Scientist
Research Engineer / Scientist

Apple, Inc. • Cupertino (CA)

Sur place
USD 216 000 - 325 000
Stock plan
Relocation assistance
Education reimbursement
AIML - Data Scientist, Evaluation
AIML - Data Scientist, Evaluation

Apple, Inc. • Cupertino (CA)

Sur place
USD 150 000 - 278 000
Stock programs
Medical and dental coverage
Retirement benefits
+4
AIML - Data Scientist, Evaluation
AIML - Data Scientist, Evaluation

Apple, Inc. • Cupertino (CA)

Sur place
USD 147 400 - 272 100
Employee stock programs
Comprehensive medical coverage
Retirement benefits
+1