AIML - Sr. LLM Inference Quality Engineer

Apple, Inc.

Seattle (WA)

On-site

USD 140,000 - 190,000

Full time

6 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Apple is seeking a senior AI systems engineer to own LLM inference correctness, consistency, and stability in production. You will design and build automation frameworks, evaluation methodologies, and qualification strategies for models, prompts, serving stacks, and infrastructure changes before they reach users.

You will lead qualification for generative AI workloads, write production-quality code in Python/C++/Swift, and collaborate with ML engineers, inference engineers, and platform teams to

Qualifications

  • 5+ years in hardware/software development lifecycle processes.
  • Ability to own LLM inference quality in production and define qualification strategies.

Responsibilities

  • Design and develop qualification frameworks for LLM inference at data center scale.
  • Build the LLM inference output-correctness automation framework for large-scale deployment.
  • Architect end-to-end test/qualification strategy for distributed ML serving systems.
  • Design harnesses for validation of ML/LLM services with mocking and synthetic traffic.
  • Gate releases in CI: ensure model/weight/prompt/runtime changes are evaluated pre-release.
  • Create regression suites tracking quality across versions over time.
  • Validate tool-use, function-calling, and multi-turn agentic workflows for correctness.
  • Develop safety, refusal, and policy-compliance coverage across releases.
  • Probe AI-specific risks: prompt injection, data leakage, hallucinations, latency.
  • Build adversarial/robustness tests, long-context and ambiguous inputs.
  • Qualify inference-optimization techniques for output parity with full-precision references.
  • Validate output equivalence across serving backends and accelerators.
  • Perform comprehensive testing including performance, fault-injection, reliability.
  • Develop AI QA tooling for automated test generation and defect triage.
  • Define metrics, logging, tracing, and observability for production monitoring.
  • Apply log analysis to root-cause failures in complex LLM serving systems.
  • Conduct regression and integration testing across platforms including iOS and macOS.
  • Develop production AI infrastructure software in Python/C++/Swift with good modularity and performance.

Skills

LLM inference experience
Automation framework design
Quality engineering
Statistical ML evaluation
Production systems ownership
Python
C++
Swift
Data-center scale testing

Tools

Python
C++
Swift
Splunk
CI/CD tooling

Job description

Apple is building the next generation of AI data center infrastructure to power large language model (LLM) services at scale. We are looking for a senior engineer to own the correctness, consistency, and stability of LLM inference in production. You will design and build the automation frameworks, evaluation methodology, and qualification strategy that give us confidence in every model, prompt, serving stack, and infrastructure change before it reaches our users. Your work is the safety net that lets Apple ship frontier AI quickly, without sacrificing quality, safety, or trust.

Description

In this role, you will lead the qualification of an LLM inference, output-correctness, automation framework for large-scale AI data center deployment and platform validation. You will architect qualification systems for generative AI, including agentic workflows and tool-using models, running at data center scale, validating that outputs remain correct, consistent, and stable as models, weights, prompts, serving runtimes, quantization schemes, and accelerator hardware change underneath them.

This is a hands-on AI systems engineering role at the intersection of machine learning, distributed systems, and quality engineering. You will write production-quality software in Python, C++, and/or Swift, apply statistical and ML-evaluation methods to validate both data and model behavior, and build AI-assisted QA tooling that makes testing itself smarter and faster. You will partner closely with ML engineers, inference/performance engineers, distributed systems engineers, and platform teams, and you will be a trusted Quality voice on what "correct" and "stable" mean for LLM inference at scale, including the inherently non-deterministic, probabilistic nature of generative output. You should be comfortable operating in ambiguity, defining quality bars where none exist yet, and delivering rigorous results on tight schedules.

Responsibilities
  • Design and develop qualification frameworks for LLM inference systems deployed at data center scale, validating output correctness, consistency, and stability across model, weight, prompt, runtime, and infrastructure changes.
  • Build the LLM inference output-correctness automation framework used for large-scale deployment and platform validation.
  • Architect the end-to-end test and qualification strategy for distributed ML serving systems - spanning unit, integration, regression, and end-to-end layers.
  • Design harnesses that enable validation of ML/LLM services independently of upstream model and service dependencies (mocking, trace replay, and synthetic traffic generation).
  • Gate releases in CI: every change to a model, weight, prompt, tool definition, or runtime runs against the relevant evals, producing a before/after comparison a reviewer can read in one glance.
  • Create prompt-level and scenario-based regression suites, and track quality across model, prompt, and infrastructure versions over time.
  • Validate tool-use, function-calling, and multi-turn agentic workflows for correct invocation, argument formation, task completion, and multi-turn coherence; flag output that is wrong, empty, or silently degraded.
  • Develop safety, refusal, and policy-compliance coverage, verifying guardrail and alignment behavior holds across releases.
  • Probe AI-specific risk: prompt injection, data leakage across users and permissions, hallucinated or ungrounded output, and cost/latency regressions.
  • Build adversarial and robustness tests, long-context, ambiguous and underrepresented inputs, to surface failure modes before users encounter them.
  • Qualify inference-optimization techniques (quantization, KV-cache, speculative decoding, batching) for output parity against a full-precision reference.
  • Validate output equivalence across serving backends and accelerators, ensuring hardware and runtime swaps preserve generation quality.
  • Perform functional, performance, fault-injection, and reliability testing, benchmarking quality-vs-latency/throughput trade-offs to surface the correctness cost of performance optimizations.
  • Develop and use AI-powered/agentic QA tooling that improves test‑case generation, defect detection, flaky‑test analysis, and failure triage automating any check performed manually more than once.
  • Define and implement metrics, logging pipelines, distributed tracing, and observability tooling to monitor output correctness, model quality, data quality, drift, latency, and system health in production.
  • Apply log analysis using Splunk and Apple internal debugging tools to root-cause failures in complex LLM serving systems.
  • Perform regression, performance, and integration testing across platforms, including iOS and macOS, for on‑device and server‑backed AI features.
  • Develop production AI infrastructure software in Python, C++, and/or Swift, applying modularity, scalability, testability, and performance principles.
Minimum Qualifications

5+ years of experience with hardware and/or software development lifecycle proces

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior LLM Inference QA & Automation Engineer
Senior LLM Inference QA & Automation Engineer

Apple, Inc. • Seattle (WA)

On-site
USD 140,000 - 190,000
AIML - Senior Machine Learning Platform Engineer, Responsible AI and Safety
AIML - Senior Machine Learning Platform Engineer, Responsible AI and Safety

Apple, Inc. • Cupertino (CA)

On-site
USD 180,000 - 240,000
Software Engineering Manager, AIML — LLM Inference Quality
Software Engineering Manager, AIML — LLM Inference Quality

Apple, Inc. • Seattle (WA)

On-site
USD 196,000 - 338,000
Medical & dental coverage
Stock programs & RSUs
Education reimbursement
+1
AIML LLM Inference Quality Engineering Manager
AIML LLM Inference Quality Engineering Manager

Apple, Inc. • Seattle (WA)

On-site
USD 196,000 - 338,000
Medical & dental coverage
Stock programs & RSUs
Education reimbursement
+1
AIML - Sr Machine Learning Platform Engineer, Evaluation
AIML - Sr Machine Learning Platform Engineer, Evaluation

Apple, Inc. • Seattle (WA)

On-site
USD 180,000 - 280,000
Data Scientist - Intelligent Experiences, MLO
Data Scientist - Intelligent Experiences, MLO

Apple, Inc. • Cupertino (CA)

On-site
USD 180,000 - 280,000
Software Engineering (AI/ML), Retail and Marcom Engineering
Software Engineering (AI/ML), Retail and Marcom Engineering

Apple, Inc. • Austin (TX)

On-site
USD 180,000 - 260,000
Agentic Systems Engineer, Proactive Intelligence — Automation & LLM Orchestration
Agentic Systems Engineer, Proactive Intelligence — Automation & LLM Orchestration

Apple, Inc. • Cupertino (CA)

On-site
USD 180,000 - 260,000
Sr. Applied Scientist, AI Evaluation & Quality Systems
Sr. Applied Scientist, AI Evaluation & Quality Systems

Apple Inc. • Seattle (WA)

On-site
USD 142,000 - 263,000
Apple Benefits
Relocation assistance
Employee stock purchase plan
+2
AIML - Sr Software Engineer - AI, Evaluation
AIML - Sr Software Engineer - AI, Evaluation

Apple, Inc. • Cupertino (CA)

On-site
USD 150,000 - 278,000
Comprehensive medical and dental
Retirement benefits
Employee stock programs
+2