AI Software Evaluation Engineer, Siri AI Quality

Apple, Inc.; Cupertino (CA); On-site; USD 150,000 - 278,000; Full time

Benefits offered by this job

Medical & dental coverage

Stock programs

Tuition reimbursement

Relocation assistance

Employee discounts

Job summary

Apple, Inc. is seeking an AI Software Evaluation Engineer for the Siri AI Quality team in Cupertino, CA.

You will help build end-to-end evaluation systems for Siri AI experiences, turning results into actionable insights and partnering with product, engineering, evaluator, persona and data-platform teams to raise quality across platforms. The role emphasizes evaluating AI/LLM scenarios, creating datasets, metrics, and automated tooling, with opportunities to influence cross-functional product

Qualifications

• 5+ years of industry experience in software engineering, quality engineering, data engineering or ML engineering.
• 2+ years of hands-on experience building LLM-based applications or agents, including tool use, multi-step workflows and prompt engineering.
• Experience designing and running AI or LLM evaluation: building evaluation datasets, defining metrics and rubrics, LLM-as-judge or automated evaluators, and analyzing failures.
• Strong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards.
• Strong programming skills in Python and experience with automation and CI/CD.
• Strong communication and collaboration skills, with the ability to influence cross-functional partners and work across teams.

Responsibilities

• Build the systems that evaluate Siri's AI experiences and turn results into action.
• Span multiple areas: evaluation data insights, coverage analysis, and agentic scenario generation, triage and auto-fix.
• Extend automation to new domains and device platforms; collaborate with engineering and product teams to define quality standards.

Skills

5+ years exp

LLM applications

SQL & Python

Python programming

CI/CD automation

Cross-functional collaboration

Tools

MCP servers

Agent frameworks

Tableau

Snowflake

Job description

AI Software Evaluation Engineer, Siri AI Quality

Cupertino, California, United States Software and Services

Play a part in the next revolution in human-computer interaction and redefine mobile computing through voice interaction!Apple's Siri AI Quality team is looking for a passionate and skilled AI Software Evaluation Engineer to join our mission of delivering the highest-quality AI experiences to millions of users worldwide. We own the end-to-end Siri pre-ship evaluation dataset and the synthetic personas that power it. In this role, you will combine agentic engineering, data analytics and AI evaluation experience to help us catch quality issues early, automate repetitive eval operations, and expand coverage as Siri's features and platforms grow.

Description

In this role, you will build the systems that evaluate Siri's AI experiences and turn the results into action. Your work will span multiple areas of focus:* Evaluation data insights. Investigate why test scenarios regress and which issues recur. Turn failure and bug data into root-cause findings and recommendations that product, engineering and partner teams act on.* Coverage analysis. Find gaps across app intents, features and product areas, and partner with domain owners to close them. Build and report on dataset quality, issue trends and coverage.* Agentic scenario generation, triage and auto-fix. Design agentic workflows that propose, ground, validate, triage and nominate new evaluation scenarios with minimal manual effort. Extend automated screening, triage and fix tooling to new domains and device platforms.You will work with engineering, product, evaluator, persona and data-platform teams to define quality standards and deliver evaluation systems that guide Siri's development.

Minimum Qualifications

• 5+ years of industry experience in software engineering, quality engineering, data engineering or ML engineering.
• 2+ years of hands‑on experience building LLM-based applications or agents, including tool use, multi-step workflows and prompt engineering.
• Experience designing and running AI or LLM evaluation: building evaluation datasets, defining metrics and rubrics, LLM-as-judge or automated evaluators, and analyzing failures.
• Strong data analytics skills, including SQL and Python, and experience turning large datasets into clear insights and dashboards.
• Strong programming skills in Python and experience with automation and CI/CD.
• Strong communication and collaboration skills, with the ability to influence cross-functional partners and work across teams.

Preferred Qualifications

• Experience with agent frameworks and developer tooling, such as MCP servers, skills or plugins, and agentic testing.
• Experience with voice assistants, conversational AI or multi-turn dialogue evaluation.
• Experience with data schemas, validation frameworks, and structured data formats (e.g. JSON).
• Experience with data visualization and BI tools (e.g. Tableau) and with data warehouses (e.g. Snowflake).
• Familiarity with coverage analysis, traffic or taxonomy analysis, and root‑cause analysis of flaky or regressing tests.
• Comfortable working in a fast‑paced, collaborative environment with shifting priorities.

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan. You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.
Learn about accessibility in Apple’s workplace
Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.