Siri Evaluation Benchmarking Lead

Apple, Inc.; Cambridge; On-site; GBP 120,000 - 180,000; Full time

Job summary

Apple is seeking a Senior Manager to lead evaluation development and data science for Siri, directing teams across data collection, annotation, and tooling. You will shape high-impact evaluation programs across devices and platforms with privacy at the core, guiding budgets and cross-functional partnerships.

The role emphasizes scalable measurement, human-in-the-loop methodologies, and collaboration with engineering teams to advance quality metrics and data pipelines.

Qualifications

• Agentic coding proficiency for data-science tasks
• Strong understanding of metrics, crowd science, data collection and statistics
• Ability to work independently and cross-functionally in partner reporting pipelines
• Excellent communication in a collaborative environment

Responsibilities

• Design and execute efficient data collection processes using humans in the loop
• Lead annotation efforts for various languages and devices
• Plan and manage the budget of annotator resourcing with Finance, Annotation Ops, and International team partners
• Track and improve the quality of human judgments through training materials, clear questions, and efficient reviews
• Collaborate with engineering teams to build tooling for managing and browsing rich datasets

Skills

Agentic coding

Metrics & crowd science

Data collection

Annotation analysis

Statistics

Communication

Tools

Python

Data visualization

Job description

Apple is seeking a Senior Manager to lead evaluation development and data science for Siri, directing teams across data collection, annotation, and tooling. You will shape high-impact evaluation programs across devices and platforms with privacy at the core, guiding budgets and cross-functional partnerships.

The role emphasizes scalable measurement, human-in-the-loop methodologies, and collaboration with engineering teams to advance quality metrics and data pipelines.