Director, AI Platform & Inference Systems

Arm; San Jose (CA); Hybrid; USD 260,000 - 352,000; Full time

Job summary

Arm in San Jose, California, is seeking a Director/Principal Product Manager to lead the strategy and execution of a next-generation AI Platform at the intersection of hardware and software. This role defines how modern AI models are executed, served, and optimized at scale across distributed compute environments.

You will own the core platform stack, driving innovation in performance, efficiency, and scalability while partnering with engineering, research, and infrastructure teams to deliver

Qualifications

• Proven experience in product management within AI/ML infrastructure, cloud platforms, or distributed systems.
• Deep understanding of AI inference systems and large-scale serving architectures.
• Solid understanding of LLM inference concepts (prefill vs decode, KV cache, token streaming).
• Demonstrated ability to deliver scalable, high-performance platform products.
• Strong technical expertise and ability to collaborate directly with engineering teams.

Responsibilities

• Define and drive the product vision, strategy, and roadmap for large-scale AI inference systems.
• Own product direction across compute execution, inference serving, and control plane systems.
• Drive inference architecture and orchestration strategy, including distributed serving, routing, batching, and scheduling.
• Define capabilities for KV cache management, memory optimization, and token lifecycle efficiency (prefill vs decode).
• Partner with engineering to enable hardware–software co-design, improving performance across accelerators and interconnects.
• Shape the development of inference platform capabilities that deliver measurable gains in latency, throughput, and cost efficiency.
• Own and evolve control plane services (APIs, policy engines, context/state management, usage/accounting).
• Translate complex technical systems into clear product value and customer impact.
• Align multi-functional collaborators across infrastructure, research, and product organizations.
• Define and track key performance metrics (P99 latency, TTFT, throughput, cost per inference/token, availability).
• Influence platform direction across constantly evolving AI workloads (LLMs, multimodal, agentic systems).

Skills

Product mgmt

AI/ML infra

Cloud platforms

Distributed systems

LLM inference

Cross-functional collaboration

Tools

TensorRT-LLM

vLLM

Accelerators

Job description

Arm in San Jose, California, is seeking a Director/Principal Product Manager to lead the strategy and execution of a next-generation AI Platform at the intersection of hardware and software. This role defines how modern AI models are executed, served, and optimized at scale across distributed compute environments.

You will own the core platform stack, driving innovation in performance, efficiency, and scalability while partnering with engineering, research, and infrastructure teams to deliver