ML Evaluation Architect: Generative AI Quality Metrics

Apple, Inc.; Sunnyvale (CA); On-site; USD 185,000 - 325,000; Full time

Job summary

Apple, Inc. in Sunnyvale, CA, seeks an experienced evaluation scientist to shape the quality of generative imaging features across devices.

You will design evaluation sets, write guidelines, and build lightweight labeling tools for image and video review, ensuring robust human and automated judgments. You will own end-to-end quality evaluation, from measurement design to reporting model-vs-model comparisons with confidence intervals and actionable recommendations for product teams.

Qualifications

• MS and a minimum of 5 years relevant industry experience.
• 5+ years of experience in data science, machine learning evaluation, algorithm analysis, or related technical disciplines.
• Experience designing and analyzing evaluations and experiments, especially for ML or generative model quality.
• Strong Python skills and comfort with SQL, with the ability to build and maintain evaluation pipelines and small tools.
• Familiarity with AI-assisted analysis workflows, foundation models, agentic systems, or intelligent automation approaches for technical problem solving.

Responsibilities

• Build the measurement foundation. Design and maintain golden sets and evaluation sets, analyzing data distributions so results stay representative and known defects stay covered.
• Run human evaluation at quality. Write and iterate annotation guidelines and label schemas, build or adapt lightweight labeling tools for image and video review, monitor annotator quality, and resolve disagreements and guideline ambiguity.
• Automate the repeatable parts. Develop automated metrics and LLM-as-judge systems, calibrate them against human labels, and track where they diverge.
• Make the results decision-ready. Report model-vs-model comparisons with confidence intervals, significance testing, and breakdowns by hotspot, content type, and failure mode — then turn them into root-cause findings and prioritized recommendations.
• Set the bar with your partners. Work with ML, engineering, and design to define quality standards, and communicate results clearly to technical and non-technical audiences alike.

Skills

Python

SQL

Data science

ML evaluation

Experiment design

Education

MS degree

Tools

SQL

Job description

Apple, Inc. in Sunnyvale, CA, seeks an experienced evaluation scientist to shape the quality of generative imaging features across devices.

You will design evaluation sets, write guidelines, and build lightweight labeling tools for image and video review, ensuring robust human and automated judgments. You will own end-to-end quality evaluation, from measurement design to reporting model-vs-model comparisons with confidence intervals and actionable recommendations for product teams.