Staff AI Inference Runtime Engineer

Arm; Seattle (WA); Hybrid; USD 209,000 - 283,000; Full time

Benefits offered by this job

Hybrid working

Recruitment accommodations

Job summary

Arm in Seattle is seeking a Software Engineer for the AI Inference Runtime team to set technical direction for distributed inference runtime powering SOTA AI models. You will lead hands-on work across scheduling, batching, KV-cache management, memory allocation, distributed execution, kernel development and optimization, shaping how efficiently models use compute.

You will partner with AI Infrastructure, compute and product teams to raise performance and energy efficiency of Arm’s AI platform,

Qualifications

• 5+ years in ML systems, high-performance systems, compilers, kernel development, or production AI inference.
• Deep understanding of modern AI inference, including model execution, Attention, MoE, batching, prioritisation, and KV-cache behavior.
• Strong programming in C++, Rust, Python with concurrency and memory management.
• Proven ability to profile, debug, and optimize performance across kernels, runtimes, frameworks, and hardware.

Responsibilities

• Define architecture, interfaces, and roadmap for AI inference runtime capabilities.
• Enable model architectures end to end through operator support, validation, and optimization of scheduling, batching, and memory management.
• Profile bottlenecks and develop optimized kernels and data movement paths across compute, memory, networking, and framework integration.
• Evaluate new inference techniques and build benchmarking, regression, validation, and safe-rollout systems.
• Partner with cloud, framework, compiler, hardware, and research teams; mentor engineers and drive performance-engineering practices.

Skills

ML systems

high-performance systems

compilers

kernel development

performance optimization

concurrency & parallelism

memory management

profiling & debugging

Tools

C++

Rust

Python

Assembler/Intrinsics

Job description

Arm in Seattle is seeking a Software Engineer for the AI Inference Runtime team to set technical direction for distributed inference runtime powering SOTA AI models. You will lead hands-on work across scheduling, batching, KV-cache management, memory allocation, distributed execution, kernel development and optimization, shaping how efficiently models use compute.

You will partner with AI Infrastructure, compute and product teams to raise performance and energy efficiency of Arm’s AI platform,