Director of AI Evaluation & Quality Assurance

Apple, Inc.; Cupertino (CA); On-site; USD 311,000 - 497,000; Full time

Benefits offered by this job

Apple stock programs

Discretionary bonuses or commission

Discretionary employee stock purchase

Comprehensive medical and dental

Retirement benefits

Relocation assistance

Job summary

Apple in Cupertino, CA seeks a Director to build and lead the evaluation and quality assurance function for a major multi-year engineering initiative. You will define end-to-end quality, create scale-ready evaluation infrastructure, and recruit the team, tooling, and operating rhythm needed to catch issues before users are affected.

You will partner with engineering, applied ML, and program management leadership, setting strategy and direction with latitude in an area with no existing

Qualifications

• 12+ years of experience in data science, machine learning evaluation, quality assurance, or test engineering roles in a technical product environment, including experience building and leading multi-level QA or evaluation organizations through managers.
• Demonstrated experience building an evaluation or QA function from the ground up — defining metrics, process, tooling, and organizational structure where none previously existed.
• Extensive experience utilizing LLM-based autograders and rubrics, including rubric design, evaluation set development, validation, calibration, and human-model alignment, and using them to proactively drive product and model improvements.
• Deep experience in traditional human evaluation methodology, logging, and A/B testing for consumer-facing products at scale.
• Experience evaluating conversational AI, recommendation, ranking, or search systems, and — ideally — agentic or tool-use systems that take real-world actions, not just generate responses.
• BS in Computer Science, Engineering, or a related technical field, or equivalent experience.
• Experience with quality assurance for third-party API integrations.
• Experience with logging infrastructure and instrumentation for AI product quality measurement.
• Exceptional written and verbal communication skills, with a track record of presenting quality status, risk, and trade-offs clearly to, and influencing decisions among, senior executive leadership.

Responsibilities

• Design metrics and automated grading systems ("autograders") that scale to large, frequently changing datasets, utilizing those alongside human evaluation, and determining where the line between those should sit.
• Wiring evaluation results into release processes as explicit quality gates, not just after-the-fact reporting.
• Catching and preventing quality regressions across an entire product surface — from user-facing correctness to backend transaction integrity.
• Partnering with engineering and applied ML teams to define what "quality" and "done" mean before a product's own definitions of those things have settled.
• Navigating ambiguity and building structure — process, tooling, org design — in an environment where none of it exists yet.
• Making judgment calls under uncertainty about where to invest limited evaluation resources for maximum risk reduction.

Skills

Data science

Machine learning evaluation

Quality assurance

Test engineering

Leadership

Education

BS in Computer Science or Engineering

Tools

LLM-based autograders

A/B testing tooling

Job description

Apple in Cupertino, CA seeks a Director to build and lead the evaluation and quality assurance function for a major multi-year engineering initiative. You will define end-to-end quality, create scale-ready evaluation infrastructure, and recruit the team, tooling, and operating rhythm needed to catch issues before users are affected.

You will partner with engineering, applied ML, and program management leadership, setting strategy and direction with latitude in an area with no existing