AI Software Evaluation Engineer, Siri AI Quality

Apple, Inc.

Cupertino (CA)

On-site

USD 180,000 - 260,000

Full time

10 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Apple's Siri AI Quality team is seeking an AI Software Evaluation Engineer to build systems that evaluate Siri's AI experiences and guide product improvement from data.

You will analyze test failures, drive coverage across intents and devices, and design agentic workflows with automated screening and triage. Strong Python/SQL skills and experience with LLM evaluation are essential.

Qualifications

  • 5+ years of experience in software/quality/data/ML engineering.
  • 2+ years building LLM-based applications or agents with tool use and prompt engineering.
  • Experience designing and running AI/LLM evaluation and analyzing failures.
  • Strong data analytics with SQL and Python and turning datasets into insights.
  • Strong Python programming and automation/CI/CD experience.
  • Excellent communication and collaboration across teams.

Responsibilities

  • Evaluate Siri AI experiences and turn results into actions.
  • Investigate test regressions and derive root causes for product teams.
  • Develop coverage analyses across intents/features and report dataset quality.
  • Design agentic workflows for evaluation scenarios and expand tooling across domains.

Skills

LLM evaluation
SQL
Python
Prompt engineering
Data analytics
CI/CD
Communication

Tools

Tableau
Snowflake
MCP servers
Agentic testing tooling

Job description

Play a part in the next revolution in human-computer interaction and redefine mobile computing through voice interaction!

Apple's Siri AI Quality team is looking for a passionate and skilled AI Software Evaluation Engineer to join our mission of delivering the highest-quality AI experiences to millions of users worldwide. We own the end-to-end Siri pre-ship evaluation dataset and the synthetic personas that power it. In this role, you will combine agentic engineering, data analytics and AI evaluation experience to help us catch quality issues early, automate repetitive eval operations, and expand coverage as Siri's features and platforms grow.

In this role, you will build the systems that evaluate Siri's AI experiences and turn the results into action. Your work will span multiple areas of focus:

  • * Evaluation data insights. Investigate why test scenarios regress and which issues recur. Turn failure and bug data into root-cause findings and recommendations that product, engineering and partner teams act on.
  • * Coverage analysis. Find gaps across app intents, features and product areas, and partner with domain owners to close them. Build and report on dataset quality, issue trends and coverage.
  • * Agentic scenario generation, triage and auto-fix. Design agentic workflows that propose, ground, validate, triage and nominate new evaluation scenarios with minimal manual effort. Extend automated screening, triage and fix tooling to new domains and device platforms.

You will work with engineering, product, evaluator, persona and data-platform teams to define quality standards and deliver evaluation systems that guide Siri's development.

Minimum Qualifications
  • 5+ years of industry experience in software engineering, quality engineering, data engineering or ML engineering.
  • 2+ years of hands-on experience building LLM-based applications or agents, including tool use, multi-step workflows and prompt engineering.
  • Experience designing and running AI or LLM evaluation: building evaluation datasets, defining metrics and rubrics, LLM-as-judge or automated evaluators, and analyzing failures.
  • Strong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards.
  • Strong programming skills in Python and experience with automation and CI/CD.
  • Strong communication and collaboration skills, with the ability to influence cross-functional partners and work across teams.
Preferred Qualifications
  • Experience with agent frameworks and developer tooling, such as MCP servers, skills or plugins, and agentic testing.
  • Experience with voice assistants, conversational AI or multi-turn dialogue evaluation.
  • Experience with data schemas, validation frameworks, and structured data formats (e.g. JSON).
  • Experience with data visualization and BI tools (e.g. Tableau) and with data warehouses (e.g. Snowflake).
  • Familiarity with coverage analysis, traffic or taxonomy analysis, and root-cause analysis of flaky or regressing tests.
  • Comfortable working in a fast-paced, collaborative environment with shifting priorities.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Engineering Program Manager, Siri Evaluation
Engineering Program Manager, Siri Evaluation

Apple, Inc. • Cupertino (CA)

On-site
USD 145,000 - 264,000
Comprehensive medical and dental
Employee stock programs
Relocation assistance
Sr. Machine Learning Engineer, ML Systems Evaluation Engineering
Sr. Machine Learning Engineer, ML Systems Evaluation Engineering

Apple Inc. • Cupertino (CA)

On-site
USD 185,000 - 325,000
Sr. Machine Learning Engineer, ML Systems Evaluation Engineering
Sr. Machine Learning Engineer, ML Systems Evaluation Engineering

Apple, Inc. • Seattle (WA)

On-site
USD 175,000 - 309,000
Stock awards
Employee stock purchase plan
Education reimbursement
+1
Senior Manager, Evaluation, Services Special Projects
Senior Manager, Evaluation, Services Special Projects

Apple, Inc. • Cupertino (CA)

On-site
USD 180,000 - 260,000
AIML - Sr Machine Learning Platform Engineer, Evaluation
AIML - Sr Machine Learning Platform Engineer, Evaluation

Apple, Inc. • Seattle (WA)

On-site
USD 180,000 - 280,000
Senior Engineering Manager, Multilingual AI - Siri
Senior Engineering Manager, Multilingual AI - Siri

Apple, Inc. • Cupertino (CA)

On-site
USD 300,000 - 460,000
AIML - Sr Software Engineer - AI, Evaluation
AIML - Sr Software Engineer - AI, Evaluation

Apple, Inc. • Cupertino (CA)

On-site
USD 150,000 - 278,000
Comprehensive medical and dental
Retirement benefits
Employee stock programs
+2
Sr. Machine Learning Engineer, Speech LLM Evaluation
Sr. Machine Learning Engineer, Speech LLM Evaluation

Apple, Inc. • Cambridge (MA)

On-site
USD 166,000 - 292,000
Stock programs
RSU awards
Employee stock plan discount
+3
Machine Learning Engineer, Siri Speech
Machine Learning Engineer, Siri Speech

Apple, Inc. • Cupertino (CA)

On-site
USD 129,000 - 225,000
Medical and dental coverage
Employee stock programs
Tuition reimbursement
+1
AIML - Sr Data Scientist, Evaluation
AIML - Sr Data Scientist, Evaluation

Apple, Inc. • Cupertino (CA)

On-site
USD 185,000 - 325,000
Medical and dental coverage
Stock program eligibility
Tuition reimbursement
+3