Staff Software Engineer, Inference Runtime Lead

Anthropic; San Francisco (CA); Hybrid; USD 405,000 - 485,000; Full time

Job summary

Anthropic’s Inference organization seeks a Staff Engineer to be the technical lead for Inference Runtime—the shared, accelerator‑agnostic core of our inference serving stack. You will set the architecture, roadmap, and validation workflows for a system used across GPUs, TPUs, and Trainium.

You’ll mentor engineers, partner with the Infrastructure org, and own performance and correctness requirements, balancing cross‑platform tradeoffs while driving the team toward scalable abstractions as the

Qualifications

• Deep background in systems engineering or ML infrastructure with hands-on performance profiling and debugging at scale.
• Significant experience with at least one accelerator ecosystem (CUDA/GPUs, TPU, or Trainium/AWS Neuron) and runtime agnosticism.
• Extensive software engineering experience in large-scale distributed systems serving millions of users.
• Experience defining and using engineering metrics to drive improvements (SLOs, escape rates, latency or throughput).
• Ability to drive technical alignment across teams and communicate effectively without formal authority.

Responsibilities

• Set technical direction, owning architecture and roadmap for the shared runtime of the inference serving stack.
• Own and evolve the accelerator-agnostic runtime interfaces, boundaries, and build structure with hands-on work in Rust and Python.
• Keep expansion costs low by aligning models and deployment targets with core capabilities.
• Drive efficient accelerator usage across GPU, TPU, and Trainium platforms.
• Build validation surfaces with partitioned builds, canary/shadow/rollback mechanisms.
• Collaborate with Infrastructure on compilers, build systems, and toolchains to meet performance requirements.
• Mentor engineers through design/code reviews while raising technical standards.

Skills

Systems engineering

Performance profiling

Latency optimization

Throughput optimization

Distributed systems

SLOs & metrics

Cross-org collaboration

Communication

Education

Bachelor's degree in Computer Science or related field

Tools

CUDA/GPU

TPU

Trainium/AWS Neuron

Rust

Python

Kubernetes

CI/CD

Job description

Anthropic’s Inference organization seeks a Staff Engineer to be the technical lead for Inference Runtime—the shared, accelerator‑agnostic core of our inference serving stack. You will set the architecture, roadmap, and validation workflows for a system used across GPUs, TPUs, and Trainium.

You’ll mentor engineers, partner with the Infrastructure org, and own performance and correctness requirements, balancing cross‑platform tradeoffs while driving the team toward scalable abstractions as the