ML Evaluation Scientist: Foundation & Multimodal Models

Apple Inc.; Sunnyvale (CA); On-site; USD 150,400 - 277,600; Full time

Benefits offered by this job

Comprehensive medical and dental cover

Retirement benefits

Relocation assistance

Job summary

Apple Inc. is seeking an expert in evaluating machine learning and deep learning models, including foundation models and multimodal systems.

You will craft robust evaluation frameworks, using traditional statistics and LLMs as judges to assess tasks like summarization and multimodal generation. The role requires strong Python expertise and a deep understanding of statistical methods, data quality, and model robustness, collaborating with ML engineers, data scientists, and ML infrastructure teams

Qualifications

• BS and a minimum of 3 years relevant industry experience.
• Strong experience evaluating supervised, unsupervised, and deep learning models.
• Hands-on experience evaluating LLMs and using them as scoring/judging mechanisms.
• Familiarity with multimodal models and related evaluation challenges.
• Proficiency in Python and libraries such as NumPy, pandas, scikit-learn, PyTorch, or TensorFlow.
• Solid understanding of statistical testing, sampling, confidence intervals, and metrics (e.g., precision/recall, BLEU, ROUGE, FID).
• Strong documentation skills, including the ability to write technical reports and present to non-technical audiences.

Responsibilities

• Develop robust methodologies to assess foundation models across diverse tasks.
• Leverage LLMs as judges for subjective evaluations and multimodal tasks.
• Build, curate, and lead evaluation datasets and benchmarks.
• Collaborate with research, engineering, and product teams to align goals with user experience.
• Conduct failure analysis to improve model robustness and document findings.

Skills

Python

NumPy

pandas

scikit-learn

PyTorch

TensorFlow

statistical testing

confidence intervals

BLEU/ROUGE/FID

documentation

Education

BS in a relevant field

Tools

OpenEval

ELO-based ranking

LLM-as-a-Judge frameworks

Job description

Apple Inc. is seeking an expert in evaluating machine learning and deep learning models, including foundation models and multimodal systems.

You will craft robust evaluation frameworks, using traditional statistics and LLMs as judges to assess tasks like summarization and multimodal generation. The role requires strong Python expertise and a deep understanding of statistical methods, data quality, and model robustness, collaborating with ML engineers, data scientists, and ML infrastructure teams