Director, Product Management

Arm; San Jose (CA); Hybrid; USD 260,000 - 352,000; Full time

Job summary

Arm in San Jose, California, is seeking a Director/Principal Product Manager to lead the strategy and execution of a next-generation AI Platform at the intersection of hardware and software. This role defines how modern AI models are executed, served, and optimized at scale across distributed compute environments.

You will own the core platform stack, driving innovation in performance, efficiency, and scalability while partnering with engineering, research, and infrastructure teams to deliver

Qualifications

• Proven experience in product management within AI/ML infrastructure, cloud platforms, or distributed systems.
• Deep understanding of AI inference systems and large-scale serving architectures.
• Solid understanding of LLM inference concepts (prefill vs decode, KV cache, token streaming).
• Demonstrated ability to deliver scalable, high-performance platform products.
• Strong technical expertise and ability to collaborate directly with engineering teams.

Responsibilities

• Define and drive the product vision, strategy, and roadmap for large-scale AI inference systems.
• Own product direction across compute execution, inference serving, and control plane systems.
• Drive inference architecture and orchestration strategy, including distributed serving, routing, batching, and scheduling.
• Define capabilities for KV cache management, memory optimization, and token lifecycle efficiency (prefill vs decode).
• Partner with engineering to enable hardware–software co-design, improving performance across accelerators and interconnects.
• Shape the development of inference platform capabilities that deliver measurable gains in latency, throughput, and cost efficiency.
• Own and evolve control plane services (APIs, policy engines, context/state management, usage/accounting).
• Translate complex technical systems into clear product value and customer impact.
• Align multi-functional collaborators across infrastructure, research, and product organizations.
• Define and track key performance metrics (P99 latency, TTFT, throughput, cost per inference/token, availability).
• Influence platform direction across constantly evolving AI workloads (LLMs, multimodal, agentic systems).

Skills

Product mgmt

AI/ML infra

Cloud platforms

Distributed systems

LLM inference

Cross-functional collaboration

Tools

TensorRT-LLM

vLLM

Accelerators

Job description

Job ID 2026-17305 Date posted 26/10/2026 Location San Jose, California Category Product Management

The future isn’t just about technology - it’s about people and the limitless possibilities AI can unlock for us all. As AI transforms how we live, work, and connect, Arm is at its very foundation, powering the innovations that are shaping our world.

Arm’s Cloud AI team is building next-generation platforms to power the AI workloads of tomorrow, and we’re growing our product management organization to help guide this journey. This is an opportunity to help shape the product direction of a major strategic investment while also helping build a new team and culture.

About The Role

We are seeking a Director / Principal Product Manager to lead the strategy and execution of a next-generation AI Platform. This role sits at the intersection of hardware and software, defining how modern AI models are executed, served, and optimized at scale across distributed compute environments.

You will own the core platform stack—driving innovation in performance, efficiency, and scalability while partnering closely with engineering, research, and infrastructure teams to deliver world-class AI systems.

Responsibilities

• Define and drive the product vision, strategy, and roadmap for large-scale AI inference systems
• Own product direction across compute execution, inference serving, and control plane systems
• Drive inference architecture and orchestration strategy, including distributed serving, routing, batching, and scheduling
• Define capabilities for KV cache management, memory optimization, and token lifecycle efficiency (prefill vs decode)
• Partner with engineering to enable hardware–software co-design, improving performance across accelerators and interconnects
• Shape the development of inference platform capabilities that deliver measurable gains in latency, throughput, and cost efficiency
• Own and evolve control plane services (APIs, policy engines, context/state management, usage/accounting)
• Translate complex technical systems into clear product value and customer impact
• Align multi-functional collaborators across infrastructure, research, and product organizations
• Define and track key performance metrics (P99 latency, TTFT, throughput, cost per inference/token, availability)
• Influence platform direction across constantly evolving AI workloads (LLMs, multimodal, agentic systems)

Required Skills And Experience

• Proven experience in product management within AI/ML infrastructure, cloud platforms, or distributed systems
• Deep understanding of AI inference systems and large-scale serving architectures
• Solid understanding of LLM inference concepts (prefill vs decode, KV cache, token streaming)
• Demonstrated ability to deliver scalable, high-performance platform products
• Strong technical expertise and ability to collaborate directly with engineering teams

Nice To Have Skills And Experience

• Experience with accelerator-based systems and performance optimization
• Familiarity with modern inference frameworks (e.g., TensorRT-LLM, vLLM)
• Experience working on production-scale AI systems

Why Join Arm?

You will play a critical role in shaping the future of AI infrastructure—defining how next-generation models are deployed, optimized, and scaled efficiently. This is an opportunity to work at the forefront of AI systems, solving complex challenges at the intersection of performance, scalability, and usability.

Salary Range

$260,000-$351,800 per year

We value people as individuals and our dedication is to reward people competitively and equitably for the work they do and the skills and experience they bring to Arm. Salary is only one component of Arm's offering. The total reward package will be shared with candidates during the recruitment and selection process.

Our 10x Mindset

At Arm, we believe progress happens when people are empowered to think bigger and push beyond what seems possible. Our 10x mindset is about curiosity, ambition and creating impact that will be used by millions. We learn fast, prioritise collaboration and turn bold ideas into real technology. We look for people who are inspired by this way of working and want to grow in an environment where bold ideas are welcomed. Discover how the 10x mindset shapes life at Arm:

Use of AI in Your Application

At Arm, we embrace the benefits AI can bring, including in recruitment. You may use AI to support your preparation, such as improving the clarity of written responses, practicing interview questions, or learning more about our industry. However, your application and interview responses must reflect your own thinking, experience, and abilities. We want to understand your authentic perspective, creativity, and approach to problem solving. As such, you may not use AI to complete assigned tasks as part of any assessments. If you plan to use AI-supported or assistive technologies as part of a reasonable adjustment, please contact

Recruitment Adjustments and Accommodations

At Arm, we want to build extraordinary teams. If you need an adjustment or an accommodation during the recruitment process, please email

Hybrid Working

Arm’s approach to hybrid working is designed to create a working environment that supports both high performance and personal wellbeing. We believe in bringing people together face to face to enable us to work at pace, whilst recognizing the value of flexibility. Within that framework, we empower groups/teams to determine their own hybrid working patterns, depending on the work and the team’s needs. Details of what this means for each role will be shared upon application. In some cases, the flexibility we can offer is limited by local legal, regulatory, tax, or other considerations, and where this is the case, we will collaborate with you to find the best solution. Please talk to us to find out more about what this could look like for you.

Equal Opportunities

Arm is an equal opportunity employer, committed to providing an environment of mutual respect where equal opportunities are available to all applicants and colleagues. We are a diverse organization of dedicated and innovative individuals, and don’t discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.