Senior ML Evaluation Engineer for Agentic AI & Generative AI

Socket.dev; Cupertino (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Apple is seeking a Machine Learning Evaluation Engineer to design and scale evaluation systems for LLM, Generative AI, and agentic AI products. You will build datasets, tooling, and quality signals to measure accuracy, relevance, groundedness, and task completion across Commerce AI experiences.

You will collaborate with ML, software, product, QA, and data science teams to establish robust evaluation practices, develop Auto Eval capabilities, and integrate them into development pipelines for

Qualifications

• Typically requires a minimum of 7 years of related experience in ML engineering, ML evaluation, software eng., data science, or a related field.
• Experience designing automated ML evaluation frameworks, metrics, benchmarks, datasets, or experimentation methodologies.
• Strong programming skills in Python and production-quality software development.
• Understanding of modern LLM architectures, prompting, retrieval, and agentive workflows.
• Experience with HITL evaluation, data-quality workflows, and statistical analysis.

Responsibilities

• Design and build scalable evaluation systems for LLM and Generative AI products.
• Define methodologies and quality metrics across accuracy, relevance, groundedness, and task completion.
• Build and maintain evaluation datasets including golden datasets and regression suites.
• Develop Auto Eval capabilities for evaluating models, prompts, and retrieval systems.
• Design model-based evaluation approaches such as LLM-as-a-Judge and calibration methods.
• Develop HITL evaluation approaches for subjective quality dimensions.
• Create evaluation rubrics, annotation guidelines, and quality standards with cross-functional teams.
• Calibrate automated evaluators against human judgment and measure reliability.
• Evaluate end-to-end AI systems including context construction, prompts, and tool use.
• Develop evaluation for multi-turn conversations, personalization, recommendations, and agentic tasks.
• Perform detailed error analysis to identify model and data improvements.
• Build reusable evaluation infrastructure, APIs, dashboards, and developer tooling.
• Integrate evaluation into CI/CD workflows and release-readiness assessments.
• Connect offline evaluation results with production signals to improve coverage.
• Collaborate with ML, Software, Product, Quality, and Data Science teams throughout lifecycle.

Skills

Python

ML Engineering

Evaluation frameworks

Data pipelines

Production software

LLMs

Generative AI

Education

Bachelor's degree in CS/ML/AI

Master's degree preferred

Tools

CI/CD pipelines

Job description

Apple is seeking a Machine Learning Evaluation Engineer to design and scale evaluation systems for LLM, Generative AI, and agentic AI products. You will build datasets, tooling, and quality signals to measure accuracy, relevance, groundedness, and task completion across Commerce AI experiences.

You will collaborate with ML, software, product, QA, and data science teams to establish robust evaluation practices, develop Auto Eval capabilities, and integrate them into development pipelines for