Senior Manager, Evaluation, Services Special Projects

Apple, Inc.; Cupertino (CA); Sur place; USD 180 000 - 260 000; Plein temps

Résumé du poste

Apple, Inc. is seeking a hands-on Senior Manager of Evaluation for Services Special Projects in Cupertino, CA.

You will build and lead a team of applied scientists and ML engineers to measure the quality of a multi-year Apple initiative, establishing when the system is correct, relevant, and trustworthy before and after release. You will own evaluation components, including methods, datasets, and automated grading systems, and drive alignment with engineering and executive leadership.

Qualifications

• 6+ years of experience in machine learning, data science, or machine learning evaluation.
• 3+ years of direct people management of scientists or engineers.
• Track record of hiring, developing, and leading teams of applied scientists or ML engineers.
• Hands-on experience building and deploying LLM-based autograders and rubrics.
• Experience evaluating conversational AI, search, or ranking systems with offline and online metrics.
• Strong statistics knowledge and experimental design, incl. A/B testing.
• Excellent written and verbal communication with senior engineers and leaders.

Responsabilités

• Lead a team that designs and runs evaluation for a conversational experience.
• Own evaluation methods, datasets, and automated grading systems.
• Set technical direction for LLM-based autograders and rubrics.
• Partner with engineering and ML teams to diagnose failures and ship fixes.
• Grow senior contributors and recruit top talent.

Connaissances

Machine learning

Data science

Team leadership

Applied ML evaluation

Statistics

A/B testing

Communication skills

Formation

MS or PhD in Computer Science, ML, Statistics, or related field

Description du poste

Senior Manager, Evaluation, Services Special Projects

Cupertino, California, United States Software and Services

We are looking for a hands-on Evaluation leader to build and lead a team of applied scientists and machine learning engineers who measure the quality of a new, multi-year Apple initiative. You will join at the start as a key contributor to the evaluation function, working with other eval leaders to establish how the team knows the system is correct, relevant, and trustworthy before and after it ships. You will own specific components of the evaluation plan, including methods, datasets, and automated grading systems that turn product behavior into clear quality signals for engineering and executive leadership.

Description

You will lead a team that designs and runs evaluation for a conversational experience that understands what people ask for, finds and ranks the right results, and allows them to complete tasks. Quality in this product spans several layers: interpreting a request correctly, retrieving and ranking the right results, composing a grounded and accurate response, and executing actions correctly through integrations with external partners. You and your team will be key contributors to the overall evaluation plan for this service and will own the evaluation methods for the layers in your area.Within your components, you will set the technical direction for large language model (LLM) based autograders and rubrics, including rubric design, evaluation set development, validation, and calibration. You will pair these with logging and instrumentation and online experimentation, working alongside the other eval leaders who own different parts of the plan. You will build the roadmap for your components, translate the broader evaluation strategy into concrete methods and deliverables, and make sure your results turn into measurable quality gains and clear inputs to release decisions.This is a hands-on role. You will review designs and experiments, stay close to the data, and mentor a team of scientists and engineers while you recruit and grow it. You will partner with engineering, applied machine learning, quality assurance, and program management teams, and you will present quality status, risks, and trade-offs for your areas to senior leadership. The product, its architecture, and its quality bar are still taking shape, so you will operate with significant latitude and help shape what "good" and "shippable" mean as the work evolves.

We're looking for someone with a proven track record in:Building automated grading systems that scale to large, frequently changing datasets and using them to driveimprovements in models and product features.Defining metrics and evaluation sets that reflect real user behavior and predict real-world quality.Turning evaluation findings into action by partnering with engineering and applied machine learning teams to diagnose failures and ship fixes.Contributing evaluation results to release processes as explicit quality gates.Growing senior individual contributors and technical leads from within a team, and recruiting strong talent in competitive hiring markets

Minimum Qualifications

• 6+ years of experience in machine learning, data science, or machine learning evaluation, including 3+ years of
• direct people management of scientists or engineers.
• Track record of hiring, developing, and leading teams of applied scientists, data scientists, or machine learning
• Hands-on experience building and deploying LLM-based autograders and rubrics, including rubric design, evaluation
• set development, validation, and calibration.
• Experience evaluating conversational AI, search, recommendation, or ranking systems, using both offline metrics
• and online experimentation.
• Strong command of statistics and experimental design, including A/B testing and interpreting noisy metrics.
• Exceptional written and verbal communication skills, with a track record of influencing engineers and senior leaders

Preferred Qualifications

• MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent
• experience.
• Experience evaluating agentic or tool-use systems that take real-world actions, including multi-turn behavior,
• planning, and task completion.
• Experience with relevance and ranking evaluation, including graded relevance judgments and ranking metrics.
• Experience with groundedness and factuality evaluation of generated responses.
• Experience with logging infrastructure and instrumentation for AI product quality measurement.
• Experience generating and validating synthetic evaluation data.
• Experience evaluating systems that integrate with third-party partner APIs.
• Experience with privacy-preserving evaluation practices.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You'll find that idea reflected in everything here - in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.

Learn about accessibility in Apple's workplace

Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.