ML Evaluation Engineer — Adversarial Testing at Scale

Apple, Inc.; Seattle (WA); On-site; USD 142,000 - 263,000; Full time

Benefits offered by this job

Medical and dental coverage

Employee stock programs

Relocation assistance

Job summary

Apple, Inc. in Seattle, WA is seeking an ML Engineer focused on automated evaluation and adversarial design. You will design, build, and scale evaluation systems for AI features, including multi-turn conversations and agent workflows.

The role emphasizes adversarial test suites, stress testing, and alignment between automated and human evaluation. Strong Python and ML framework experience is required, with a track record of guiding evaluation across features.

Qualifications

• Bachelor’s degree in Computer Science, Machine Learning, Statistics, or related field.
• 4+ years building or extending ML evaluation systems with benchmarks and quality frameworks.
• Experience defining evaluation architecture for AI/ML with conversation/session units.
• Experience designing adversarial or red-teaming test methodologies for ML models.
• Experience with Python and ML frameworks (PyTorch, TensorFlow) in production or near-production.
• Track record of owning technical direction for evaluation across multiple features.

Responsibilities

• Define and own the automated evaluation approach for AI features across single-turn and multi-turn experiences.
• Build adversarial test suites targeting model failure modes including context loss and cascading errors.
• Develop stress test protocols to validate performance under lengthy conversations and complex tool-use.
• Ensure alignment between automated and human evaluation methods and resolve disagreements.
• Collaborate with engineering to integrate evaluation into development and release workflows.
• Scale adversarial test case generation and stress test execution with automation.
• Communicate evaluation findings and readiness assessments to cross-functional partners.

Skills

ML evaluation

Python

PyTorch

TensorFlow

Production ML

Education

Bachelor’s degree in Computer Science, Machine Learning, Statistics, or related field

Tools

LangChain

LangGraph

CrewAI

AutoGen

LangSmith

Arize

Job description

Apple, Inc. in Seattle, WA is seeking an ML Engineer focused on automated evaluation and adversarial design. You will design, build, and scale evaluation systems for AI features, including multi-turn conversations and agent workflows.

The role emphasizes adversarial test suites, stress testing, and alignment between automated and human evaluation. Strong Python and ML framework experience is required, with a track record of guiding evaluation across features.