Machine Learning Evaluation Engineer

Apple Inc.; Sunnyvale (CA); Hybrid; USD 150,000 - 278,000; Full time

Benefits offered by this job

Medical & Dental coverage

Employee stock programs

Relocation support

Education reimbursement

Job summary

Apple Inc. is seeking a Machine Learning Evaluation Engineer to design and deploy evaluation strategies for multi-modal foundation models and CV/ML algorithms.

You will build scalable evaluation pipelines, perform comprehensive failure analyses, and collaborate with hardware, software, design, and core model training teams to drive product quality. The role emphasizes data curation, metric design, automation, and leveraging GenAI tools to accelerate analysis while maintaining rigorous scientific

Qualifications

• BS and a minimum of 3 years relevant industry experience.
• 3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
• Deep understanding of core ML principles including probability, statistics, data distributions, and model bias/variance.
• Experience analyzing data quality, diversity, representativeness, and coverage gaps in large datasets.
• Applied experience employing and evaluating generative models, including prompt tuning and human-in-the-loop methodologies.
• Proven track record of defining robust metrics and designing evaluation frameworks for foundation models.

Responsibilities

• Benchmark state-of-the-art multi-modal models and develop evaluation pipelines.
• Design and implement end-to-end and component-level evaluation for text, image, and video tasks.
• Perform deep failure analysis to identify root causes in models, prompts, or downstream components.
• Create cloud-based, LLM-powered workflows and dashboards for failure analysis and reporting.
• Curate high-quality datasets and ground truth, including auto-labeling and synthetic data generation.
• Collaborate with cross-functional teams to provide actionable data-driven insights for model improvement.

Skills

ML Evaluation

Python

CV/ML fundamentals

Education

BS + 3 years experience

Tools

Cloud platforms

Job description

The Data, Analytics and Quality (DAQ) team’s core mission is to evaluate and elevate advanced sensing technologies. We collaborate closely with partners in computer vision, video engineering, and other applied ML domains to deliver high quality algorithms that power intelligent features across Apple’s products. Our evaluations and analyses inform these teams and Apple leadership throughout the entire development cycle, from early prototyping to shipping a refined product. We are seeking a Machine Learning Evaluation Engineer who will combine deep technical expertise, creativity, and systems thinking to design and deploy new evaluation strategies for multi-modal foundation models and task-specific CV/ML algorithms.

Description

As a key member of this team, you will lead the benchmarking of state-of-the‑art multi‑modal models, developing comprehensive evaluation systems that include metric design, large‑scale data preparation, and advanced failure analysis automation. Rather than focusing on core model training, you will apply your deep CV and ML expertise to rigorously assess model capabilities and translate findings into actionable improvements. You will leverage modern GenAI tools to improve engineering efficiency and accelerate analysis. You will collaborate with multidisciplinary teams across HW, SW, design, and other applied fields to ensure models meet the high quality bar necessary for exceptional customer experiences. In this role you'll be responsible for:Algorithm Evaluation & Benchmarking: Design and build comprehensive evaluation pipelines that scale across large datasets. You will be responsible for both holistic end‑to‑end system evaluation and granular component‑level testing to rigorously measure model capabilities on complex text, image, and video understanding tasks.Subjective Evaluation: Solve the challenge of developing and calibrating subjective evaluations for generative models by leveraging LLM-as-Judge and human grading techniques.Deep Failure Analysis: Identify trends in large datasets and dive deep into specific failure cases to find their root causes in models, prompts, or upstream software and algorithm components. Develop failure taxonomies to track over time.Workflow Automation: Design and develop cloud‑based, LLM‑powered workflows and dashboards to streamline failure analysis, reporting, and iterative experimentation. This is key to efficiently isolate and triage problems within complex multi‑stage algorithm stacks.Evaluation Set Curation: Refine our strategy for curating high‑quality datasets and ground truth, whether through auto‑labeling with VLMs, leveraging manual annotation teams, or generating synthetic data.Data Analysis: assess quality, diversity, representativeness, and coverage gaps across large, complex datasets to ensure evaluation sets reflect real‑world usage.Cross‑Functional Collaboration: Partner closely with the core model training teams. You will provide them with actionable, data‑driven insights and metrics to guide the next iteration of model training and fine‑tuning.

Minimum Qualifications

• BS and a minimum of 3 years relevant industry experience
• 3+ years of applied experience in Machine Learning, Computer Vision, or AI System Evaluation.
• Deep understanding of core Machine Learning principles, including probability, statistics, data distributions, and model bias/variance. Ability to apply statistical rigor to ensure evaluation metrics are meaningful and reliable.
• Experience analyzing data quality, diversity, representativeness, and coverage gaps in large datasets, and translating findings into data collection and curation strategies.
• Applied experience employing and evaluating generative models, including knowledge of prompt tuning, subjective evaluation of open‑ended outputs, and human‑in‑the‑loop methodologies.
• Proven track record of defining robust metrics/KPIs and designing rigorous evaluation frameworks for foundation models. Deep experience with custom benchmark creation, automated regression testing, and LLM/VLM-as-a-Judge methodologies.
• Strong intuition for probing ML models to discover edge cases, hallucinations, and performance bottlenecks in constrained environments. Ability to translate findings into actionable recommendations for model improvement.
• Strong proficiency in Python and building well‑designed automated pipelines, including model endpoint integration, multi‑step evaluation orchestration, and robust logging for analysis and traceability.
• Experience using AI‑powered development and analysis tools to accelerate data exploration, coding, and workflow automation, while maintaining technical rigor and reproducibility.

Preferred Qualifications

• Theoretical and practical understanding of Computer Vision (CV) and Vision‑Language Models (VLMs) in order to effectively anticipate failure modes and accurately benchmark their performance.
• Excellent written and verbal communication skills, with the ability to describe complex topics to cross‑functional teams with varying levels of technical expertise.

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan. You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong. Learn about accessibility in Apple’s workplace Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.