Senior Manager, Evaluation, Services Special Projects

Apple Inc.; Cupertino (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Apple Inc. in Cupertino, California seeks a hands-on Senior Manager, Evaluation, to build and lead a team of applied scientists and ML engineers responsible for measuring quality of a new multi-year initiative.

You will own evaluation methods, datasets, and automated graders, design LLM-based rubrics, and collaborate with engineering, QA, and PM teams to drive reliable release decisions. This leadership role emphasizes mentoring, recruitment, and shaping what 'good' and 'shippable' mean as

Qualifications

• Build automated grading systems that scale to large, frequently changing datasets.
• Define metrics and evaluation sets that reflect real user behavior and predict real-world quality.
• Turn evaluation findings into action by partnering with engineering and ML teams to diagnose failures and ship fixes.
• Contribute evaluation results to release processes as explicit quality gates.
• Grow senior individual contributors and technical leads from within a team, and recruiting strong talent in competitive hiring markets.

Responsibilities

• Lead a team designing and running evaluation for a conversational experience.
• Own evaluation methods for the layers in your area, including LLM-based autograders and rubrics.
• Set technical direction, build roadmaps, and translate strategy into concrete deliverables.
• Collaborate with engineering, applied ML, QA, and PM teams to present quality status and trade-offs to senior leadership.

Skills

People management

LLM evaluation

Experiment design

A/B testing

Statistical analysis

Communication

Applied ML

Leadership

Education

MS/PhD in CS / ML / Stats

Job description

Senior Manager, Evaluation, Services Special Projects

Cupertino, California, United States Software and Services

We are looking for a hands-on Evaluation leader to build and lead a team of applied scientists and machine learning engineers who measure the quality of a new, multi-year Apple initiative. You will join at the start as a key contributor to the evaluation function, working with other eval leaders to establish how the team knows the system is correct, relevant, and trustworthy before and after it ships. You will own specific components of the evaluation plan, including methods, datasets, and automated grading systems that turn product behavior into clear quality signals for engineering and executive leadership.

Description

You will lead a team that designs and runs evaluation for a conversational experience that understands what people ask for, finds and ranks the right results, and allows them to complete tasks. Quality in this product spans several layers: interpreting a request correctly, retrieving and ranking the right results, composing a grounded and accurate response, and executing actions correctly through integrations with external partners. You and your team will be key contributors to the overall evaluation plan for this service and will own the evaluation methods for the layers in your area.Within your components, you will set the technical direction for large language model (LLM) based autograders and rubrics, including rubric design, evaluation set development, validation, and calibration. You will pair these with logging and instrumentation and online experimentation, working alongside the other eval leaders who own different parts of the plan. You will build the roadmap for your components, translate the broader evaluation strategy into concrete methods and deliverables, and make sure your results turn into measurable quality gains and clear inputs to release decisions.This is a hands-on role. You will review designs and experiments, stay close to the data, and mentor a team of scientists and engineers while you recruit and grow it. You will partner with engineering, applied machine learning, quality assurance, and program management teams, and you will present quality status, risks, and trade-offs for your areas to senior leadership. The product, its architecture, and its quality bar are still taking shape, so you will operate with significant latitude and help shape what "good" and "shippable" mean as the work evolves.We're looking for someone with a proven track record in:Building automated grading systems that scale to large, frequently changing datasets and using them to driveimprovements in models and product features.Defining metrics and evaluation sets that reflect real user behavior and predict real-world quality.Turning evaluation findings into action by partnering with engineering and applied machine learning teams to diagnose failures and ship fixes.Contributing evaluation results to release processes as explicit quality gates.Growing senior individual contributors and technical leads from within a team, and recruiting strong talent in competitive hiring markets

Minimum Qualifications

• 6+ years of experience in machine learning, data science, or machine learning evaluation, including 3+ years of
• direct people management of scientists or engineers.
• Track record of hiring, developing, and leading teams of applied scientists, data scientists, or machine learning
• Hands-on experience building and deploying LLM-based autograders and rubrics, including rubric design, evaluation
• set development, validation, and calibration.
• Experience evaluating conversational AI, search, recommendation, or ranking systems, using both offline metrics
• and online experimentation.
• Strong command of statistics and experimental design, including A/B testing and interpreting noisy metrics.
• Exceptional written and verbal communication skills, with a track record of influencing engineers and senior leaders

Preferred Qualifications

• MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent
• experience.
• Experience evaluating agentic or tool-use systems that take real-world actions, including multi-turn behavior,
• planning, and task completion.
• Experience with relevance and ranking evaluation, including graded relevance judgments and ranking metrics.
• Experience with groundedness and factuality evaluation of generated responses.
• Experience with logging infrastructure and instrumentation for AI product quality measurement.
• Experience generating and validating synthetic evaluation data.
• Experience evaluating systems that integrate with third-party partner APIs.
• Experience with privacy-preserving evaluation practices.

This posting is not for a specific job opening and by submitting your resume you are expressing interest in being contacted about this type of role at Apple in the future.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You'll find that idea reflected in everything here - in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.
Learn about accessibility in Apple's workplace
Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.