Staff Software Engineer, Scaling Inference Systems

AI Chopping Block; San Francisco, Northern (CA, KY); Hybrid; USD 320,000 - 485,000; Full time

Job summary

Anthropic is seeking an experienced software engineer to design, build, and maintain scalable inference systems powering Claude for millions of users. You will implement resilient routing, load balancing, and deployment pipelines across cloud providers and accelerators.

You will optimize compute efficiency and support researchers by delivering high-performance infrastructure for next-generation models, collaborating across teams to ensure reliable, scalable operations from SF offices.

Qualifications

• Significant software engineering experience with distributed systems.
• Results-oriented with a bias towards flexibility and impact.
• Willingness to pick up slack, even if it goes outside your job description.
• Desire to learn more about machine learning systems and infrastructure.
• Thrive in environments where technical excellence directly drives both business results and research breakthroughs.
• Care about the societal impacts of your work.

Responsibilities

• Design, build, and maintain the distributed systems that serve Claude to millions of users worldwide.
• Develop resilient, flexible systems that adapt in real time to real world events.
• Develop intelligent request routing, load balancing, and traffic management systems across thousands of accelerators and multiple cloud providers.
• Maximize compute efficiency and optimize cost across the fleet by autoscaling and orchestrating production, research, and experimental workloads across multiple cloud providers.
• Build and operate production-grade deployment pipelines for releasing new models to users.
• Provide high-performance inference infrastructure that enables researchers to develop next-generation models.
• Integrate new AI accelerator platforms and support inference for new model architectures.

Skills

Distributed systems

High-performance software

Load balancing

Kubernetes

Python

Rust

Cloud infrastructure

Research collaboration

Societal impact awareness

Education

Bachelor's degree or equivalent

Tools

AWS

GCP

Azure

Job description

Anthropic is seeking an experienced software engineer to design, build, and maintain scalable inference systems powering Claude for millions of users. You will implement resilient routing, load balancing, and deployment pipelines across cloud providers and accelerators.

You will optimize compute efficiency and support researchers by delivering high-performance infrastructure for next-generation models, collaborating across teams to ensure reliable, scalable operations from SF offices.