Machine Learning Engineer, ML/GenAI Evaluation

Apple Inc.; San Diego (CA); On-site; USD 175,000 - 310,000; Full time

Benefits offered by this job

Apple Benefits

Stock options

Tuition reimbursement

Relocation assistance

Comprehensive medical and dental

Job summary

Apple Inc. in San Diego, California, is seeking a Machine Learning Engineer specializing in Evaluation to own the full evaluation lifecycle for Wallet features. You will design criteria, metrics, and test corpora, then sign off on model readiness before deployment to hundreds of millions of users.

You'll lead robustness, fairness, and adversarial testing, collaborate with ML Engineering, Product, Privacy, and Legal teams, and translate results into concrete product quality improvements.

Qualifications

• MS in Machine Learning, Computer Science, Statistics, Applied Mathematics, or related field preferred
• 7+ years hands-on ML evaluation experience will be considered
• 5+ years hands-on ML experience with model evaluation, offline metrics, and behavioral testing
• Strong track record designing evaluation frameworks for production ML systems including fairness and calibration

Responsibilities

• Define evaluation criteria and quality metrics for ML models powering Wallet features
• Design and maintain structured test sets covering diverse real-world scenarios
• Develop evaluation methodologies for robustness testing including distribution shift and temporal drift
• Own fairness evaluation end-to-end and gate launches on fairness criteria
• Evaluate generative outputs and assess hallucination rates and grounding
• Own model quality sign-off and final evaluations before deployment
• Synthesize results into actionable insights for product decisions
• Collaborate with ML and Quality engineers to identify failure modes early

Skills

Python

Evaluation tooling

MLflow

W&B

Education

MS in Machine Learning

PhD in CS/AI (preferred)

Tools

MLflow

W&B

Job description

Machine Learning Engineer, ML/GenAI Evaluation

San Diego, California, United States Software and Services

Would you like to contribute to Machine Learning and Generative AI technologies? Are you passionate about measuring what matters and ensuring AI systems work reliably for everyone? Do you believe that rigorous evaluation — including holding models accountable to fairness standards — is what separates great ML from good ML? We truly believe it is! We are defining what exceptional looks like for machine learning across Wallet, Payments, and Commerce. As a Machine Learning Engineer specializing in Evaluation, you will establish the evaluation criteria, metrics frameworks, and quality standards that determine when models are ready to reach hundreds of millions of users. Your judgment shapes model quality and earns the confidence to ship. You'll work at the intersection of rigorous ML science and high-impact product decisions, collaborating closely with ML Engineering, Product, Privacy, and Legal teams. This unique opportunity puts you at the center of model quality — designing adversarial test strategies, surfacing failure modes before they reach users, and owning the sign-off process that ensures Apple's financial features meet the highest bar for accuracy, robustness, and reliability.

Description

The ideal candidate is a rigorous, curious ML practitioner who believes that how you measure a model is just as important as how you train it. You think critically about what metrics actually capture, know how models break in the real world, and hold quality standards others find uncomfortably high — including on dimensions like fairness. You will own the full evaluation lifecycle for ML models across Wallet features — designing test frameworks, adversarial corpora, and benchmarks that reflect the diversity of Apple's global user base, then making the final quality call before any model ships. Your findings directly shape model development priorities and product decisions at scale.

Responsibilities

• Define evaluation criteria and quality metrics for ML models powering Wallet features
• Design and maintain structured test sets covering the full diversity of real-world scenarios — varied document formats, distributions, languages, edge cases, and adversarial inputs.
• Develop evaluation methodologies for robustness testing: distribution shift, out-of-distribution generalization, temporal drift, and aggressor scenarios
• Own fairness evaluation end-to-end — define fairness metrics appropriate to each Wallet feature, build bias test suites across protected attributes and user populations, measure disparate performance across subgroups, and gate model launches on fairness criteria with the same rigor as other conventional metrics.
• Build user persona–stratified benchmarks that reflect the breadth of Wallet's global user population across spending patterns, locales, and document types
• Evaluate generative and agentic model outputs — assessing hallucination rates, faithfulness, and groundedness using LLM-as-a-judge frameworks, human evaluation protocols, and prompt regression testing
• Own model quality sign-off — establish the launch criteria, run final evaluations, and make the call on model readiness before any feature ships
• Synthesize evaluation results into clear, actionable insights that guide model development priorities and product decisions
• Partner with ML engineers and Quality engineers to identify failure modes early in the development cycle and close the loop between evaluation findings and model improvements
• Establish and evangelize evaluation best practices across the Wallet ML team, raising the quality bar for how models are tested, monitored, and maintained post-launch

Minimum Qualifications

• M.S. in Machine Learning, Computer Science, Statistics, Applied Mathematics, or a related technical field strongly preferred.
• Bachelor's degree with 7+ years hands-on experience in ML evaluation, model quality, or applied research will be considered
• 5+ years of hands-on ML experience, with deep expertise in model evaluation, offline metrics design, and behavioral testing
• Strong track record designing evaluation frameworks for production ML systems — not just accuracy/F1, but precision-recall tradeoffs, calibration, fairness, and task-specific quality dimensions
• Creative mindset with the ability to translate standard ML evaluation metrics (F1, AUC, etc.) into utility and user trust measures
• Experience testing for distribution shift, out-of-distribution generalization, and temporal drift in real-world deployed models
• Proven ability to construct adversarial test suites, aggressor scenarios, and edge-case corpora that surface model failure modes before they reach users
• Experience with structured and semi-structured document understanding, OCR pipelines, or financial data extraction is a strong plus
• Strong programming skills in Python; fluency with evaluation tooling, data pipelines, and experiment tracking (e.g., MLflow, W&B, or equivalent)
• Excellent communication skills — ability to translate metric results into product-quality narratives for engineering and executive audiences
• Experience owning model quality sign-off in a cross-functional launch process

Preferred Qualifications

• PhD in Computer Science, Data Science, Statistics, AI/ML, or a related field.
• Experience with Bayesian or causal graph-based approaches to data generation.
• Experience with causal approaches to fairness evaluation — counterfactual fairness, causal Shapley values, or structural causal model–based bias auditing.
• Experience evaluating models under privacy constraints or on-device inference settings is a plus.
• Familiarity with confidence calibration techniques and uncertainty quantification a plus
• Background in financial services, fintech, or consumer payment products

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $175,000 and $308,500, and your base pay will depend on your skills, qualifications, experience, and location.

Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan. You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits

Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.

Learn about accessibility in Apple’s workplace

Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.