Senior AI Inference Runtime Engineer - Distributed

Arm Limited; Seattle (WA); Hybrid; USD 209,000 - 283,000; Full time

Job summary

Arm Limited is seeking a Software Engineer for the AI Inference Runtime team, driving architecture and roadmaps for distributed inference workloads and model execution across hardware platforms.

You will lead profiling, kernel optimization, and efficiency improvements, collaborating with cloud, framework, compiler, hardware, and research teams. The role involves hands-on implementation, performance engineering, and mentoring engineers within Arm's AI Platforms group.

Qualifications

• 5+ years of experience in ML systems, high-performance systems, compilers, or production AI inference.
• Deep understanding of modern AI inference, including model execution, Attention, MoE, batching, prioritisation, and KV-cache behavior.
• Strong programming skills in C++, Rust, Python, or a comparable language, with knowledge of concurrency, parallel programming, memory management, and data movement.
• Proven ability to profile, debug, and optimize performance across kernels, runtimes, frameworks, OS, and hardware.

Responsibilities

• Define the architecture, interfaces, and roadmap for AI inference runtime capabilities, including abstractions that support evolving models, workloads, and compute platforms.
• Enable new model architectures end to end through operator support, production validation, and optimization of scheduling, batching, model execution, memory management, and KV-cache efficiency.
• Profile system bottlenecks and develop optimized kernels and data-movement paths across compute, memory, networking, and framework integration.
• Evaluate new inference techniques and build benchmarking, regression, validation, and safe-rollout systems to improve latency, throughput, reliability, and resource efficiency.
• Partner with cloud, framework, compiler, hardware, and research teams; lead technical reviews, mentor engineers, and establish meticulous performance-engineering practices.

Skills

5+ years experience

In-depth AI inference

C++, Rust, Python

Profiling & optimization

Job description

Arm Limited is seeking a Software Engineer for the AI Inference Runtime team, driving architecture and roadmaps for distributed inference workloads and model execution across hardware platforms.

You will lead profiling, kernel optimization, and efficiency improvements, collaborating with cloud, framework, compiler, hardware, and research teams. The role involves hands-on implementation, performance engineering, and mentoring engineers within Arm's AI Platforms group.