Senior Software Engineer, Private Cloud Compute Inference Routing

Apple Inc.; Seattle (WA); On-site; USD 150,000 - 278,000; Full time

Job summary

Apple Inc. is seeking a Senior Software Engineer to build the next generation of inference routing for Private Cloud Compute, shaping routing and scheduling across a global fleet of servers with privacy guarantees.

You will own design decisions from architecture through production, collaborating with Siri AI, Apple Intelligence, security, and SRE teams. You’ll work on high-performance server software, network protocols, and scaling strategies to ensure low latency, high availability, and

Qualifications

• 3+ years of experience designing, building and operating large distributed systems in production.
• Knowledge of LLM inference serving, including inference request routing and cache placement.
• Experience with network protocols, load balancing or scheduling at scale.
• Experience benchmarking, debugging and tuning distributed systems, plus knowledge of Unix system internals.
• Experience with cloud deployment and observability technologies such as Kubernetes.
• Bachelor's degree in an engineering field or equivalent experience.

Responsibilities

• Design, build, and operate PCC's inference routing and request scheduling services, including global traffic management, capacity aware request placement, load balancing, and failover.
• Develop adaptive routing strategies that respond to changing demand, fleet health, and available compute capacity while maintaining latency and availability objectives.
• Build admission control, prioritization, and overload management mechanisms that keep the service responsive and allocate capacity fairly across competing workloads.
• Partner with model serving and infrastructure teams to evaluate and develop routing strategies for prefill/decode disaggregation and cache locality optimizations within PCC’s privacy architecture.
• Design network protocols and implement high-performance server software for inference request dispatch and response streaming.
• Define performance metrics and benchmark routing strategies under realistic workloads, using queueing delay, request latency, throughput, and fleet utilization to guide improvements.
• Own projects from architecture through production rollout, partnering with Siri AI, Apple Intelligence, infrastructure, security, and SRE teams to deliver new capabilities and improve reliability.

Skills

LLM inference serving

Routing & scheduling

Network protocols

Distributed systems

Kubernetes

Education

Bachelor's degree in engineering or equivalent experience

Tools

TLS

X.509

Oblivious HTTP

Job description

Senior Software Engineer, Private Cloud Compute Inference Routing

Cupertino, California, United States Software and Services

Apple is looking for a senior software engineer to build the next generation of inference routing for Private Cloud Compute, the backbone of Siri AI and Apple Intelligence. You'll design distributed systems that direct inference requests across a global fleet of inference servers, balancing latency, availability, and compute efficiency while preserving PCC's privacy guarantees. Your work will directly shape how reliably and efficiently Apple delivers cloud inference at scale. You'll join iCloud Platform Engineering team. We build the server stack behind Push Notifications, iMessage, FaceTime, iCloud, iCloud Private Relay and Private Cloud Compute. Our internet scale distributed systems deliver real-time experiences with very low latency and very high availability. They reach hundreds of millions of users across a billion devices worldwide and help them connect and share their lives securely and privately. Our work spans identity, messaging, queueing, presence, routing, relays, developer APIs and much more. We're a small, highly collaborative team with a large impact.

Description

Routing LLM inference at Apple's scale requires systems that adapt to changing demand, available capacity, and failures while maintaining low latency and high utilization. PCC adds a fundamental architectural requirement: request content remains encrypted outside trusted compute nodes. You'll develop routing and scheduling strategies within this privacy model, working with inference and security teams to evaluate techniques such as cache aware request placement and prefill/decode disaggregation. You'll design system architectures, specify custom network protocols and write high performance code. You'll also build scaling strategies for services used by hundreds of millions of people. Our team is small and highly visible, so your chances to make an impact are wide open. If you take opportunities like that and run with them, you'll enjoy working here. You'll own technical decisions from early design through production, using benchmarks and operational evidence to evaluate tradeoffs. You'll work closely with model serving, infrastructure, security, and SRE engineers to turn new inference requirements into reliable routing capabilities.

Responsibilities

• Design, build, and operate PCC's inference routing and request scheduling services, including global traffic management, capacity aware request placement, load balancing, and failover.
• Develop adaptive routing strategies that respond to changing demand, fleet health, and available compute capacity while maintaining latency and availability objectives.
• Build admission control, prioritization, and overload management mechanisms that keep the service responsive and allocate capacity fairly across competing workloads.
• Partner with model serving and infrastructure teams to evaluate and develop routing strategies for prefill/decode disaggregation and cache locality optimizations within PCC’s privacy architecture.
• Design network protocols and implement high-performance server software for inference request dispatch and response streaming.
• Define performance metrics and benchmark routing strategies under realistic workloads, using queueing delay, request latency, throughput, and fleet utilization to guide improvements.
• Own projects from architecture through production rollout, partnering with Siri AI, Apple Intelligence, infrastructure, security, and SRE teams to deliver new capabilities and improve reliability.

Minimum Qualifications

• 3+ years of professional experience designing, building and operating large distributed systems in production
• Knowledge of LLM inference serving, including inference request routing, KV-cache placement and reuse (e.g., prefix-aware routing), prefill/decode disaggregation, and the tradeoffs between latency, throughput and accelerator use
• Experience with network protocols, application level networking and client-server architectures, including request routing, load balancing or scheduling at scale
• Experience benchmarking, debugging and tuning the performance of distributed systems, plus knowledge of Unix system internals
• Experience with cloud deployment and observability technologies such as Kubernetes
• Bachelor's degree in an engineering related field or equivalent experience

Preferred Qualifications

• Experience building LLM inference serving or routing systems such as llm-d, NVIDIA Dynamo, vLLM, SGLang or similar
• Familiarity with KV cache aware routing, prefix caching, prefill/decode disaggregation and accelerator capacity placement
• Experience with TLS, X.509 certificate evaluation or other secure protocols, and awareness of recent developments in standard protocols
• Understanding of cryptographic protocols, encryption, key management and privacy-preserving system design (e.g., Oblivious HTTP, remote attestation)
• Experience leading a substantial engineering project from design through production, resolving technical tradeoffs and coordinating across teams.
• Ability to explain technical tradeoffs, collaborate across engineering teams, and make progress on problems with incomplete requirements.

At Apple, base pay is one part of our total compensation package and is determined within a range. This provides the opportunity to progress as you grow and develop within a role. The base pay range for this role is between $150,400 and $277,600, and your base pay will depend on your skills, qualifications, experience, and location.
Apple employees also have the opportunity to become an Apple shareholder through participation in Apple’s discretionary employee stock programs. Apple employees are eligible for discretionary restricted stock unit awards, and can purchase Apple stock at a discount if voluntarily participating in Apple’s Employee Stock Purchase Plan. You’ll also receive benefits including: Comprehensive medical and dental coverage, retirement benefits, a range of discounted products and free services, and for formal education related to advancing your career at Apple, reimbursement for certain educational expenses — including tuition. Additionally, this role might be eligible for discretionary bonuses or commission payments as well as relocation. Learn more about Apple Benefits
Note: Apple benefit, compensation and employee stock programs are subject to eligibility requirements and other terms of the applicable plan or program.

Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant

At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.
Learn about accessibility in Apple’s workplace
Learn about reasonable accommodations for job applicants

Apple accepts applications to this posting on an ongoing basis.