AI Evaluations Architect, Decision Intelligence

Apple Inc.; Cupertino (CA); On-site; USD 185,000 - 278,000; Full time

Benefits offered by this job

Medical insurance

Dental coverage

Retirement benefits

Discounted products

Free services

Tuition reimbursement

Job summary

Apple Inc. in Cupertino, CA is seeking an AI Evaluations Engineer to own the end-to-end evaluation pipeline for AI products and agentic workflows. You will build and operate evaluation frameworks, instrumentation, and rubrics to measure accuracy, relevance, and latency across LLM outputs and agent actions.

The role requires deep experience with AI evaluation techniques, multiple LLM ecosystems, and collaboration with cross-functional teams to drive product quality and reliability.

Qualifications

• 5+ years in data and AI-related fields such as AI engineering or data science.
• Experience with AI evaluation techniques like rubric-based scoring.
• Familiarity with multiple LLM ecosystems (OpenAI, Anthropic, Gemini).
• SQL proficiency and data analytics platform exposure (Hadoop/Spark/Snowflake).
• CI/CD or release validation experience and cross-team collaboration.

Responsibilities

• Architect end-to-end AI evaluation frameworks and AB testing strategies.
• Build and operate evaluation workflows for LLM outputs across tasks.
• Define rubric-based scoring for correctness, relevance, grounding, and consistency.
• Instrument traces, prompts, responses, and user feedback for observability.
• Own the eval gate and pass/fail contracts for releases.
• Collaborate with AI engineers and platform teams to map requirements to criteria.

Skills

AI evaluation

SQL

CI/CD

Langfuse

Data analysis

Cross-functional

Education

B.S. in CS/Engineering

MS/PhD preferred

Tools

Pinecone

FAISS

Milvus

PostgreSQL

RabbitMQ

Redis

Valkey

Langfuse

Job description

Apple Inc. in Cupertino, CA is seeking an AI Evaluations Engineer to own the end-to-end evaluation pipeline for AI products and agentic workflows. You will build and operate evaluation frameworks, instrumentation, and rubrics to measure accuracy, relevance, and latency across LLM outputs and agent actions.

The role requires deep experience with AI evaluation techniques, multiple LLM ecosystems, and collaboration with cross-functional teams to drive product quality and reliability.