Distributed RL Systems Engineer - Performance & Reliability

Anthropic; San Francisco (CA); Hybrid; USD 500,000 - 850,000; Full time

Benefits offered by this job

Competitive compensation

Equity donation matching

Generous vacation & parental leave

Flexible working hours

Office in San Francisco

Job summary

Anthropic in San Francisco is seeking a Research Engineer to join the Distributed Systems team within RL Engineering. You will work on scheduling, data movement, autoscaling, and observability for large RL workloads, designing robust, scalable infrastructure.

The role emphasizes principled problem solving, cross-functional collaboration with researchers and performance engineers, and building systems that keep progressing despite hardware and network challenges.

Qualifications

• Strong software engineering skills in Python and at least one systems language.
• Experience designing, building, and operating large-scale distributed systems in production.
• Deep understanding of distributed systems fundamentals, including consistency, coordination, consensus, failure modes, and recovery.
• Ability to reason quantitatively about throughput, latency, and resource costs across compute, memory, storage, and network.
• Experience debugging complex failures across many hosts and services, including failures you can’t reproduce locally.
• Strong written communication, including design documents and incident writeups.

Responsibilities

• Design, build, and operate the distributed systems that run RL at scale, across training, sampling, and environment execution.
• Find and remove whatever currently limits the system, whether it’s scheduling, data movement, storage, networking, or coordination.
• Build fault tolerance into every layer: failure detection, isolation, and recovery that keep long-running jobs making progress without human intervention.
• Design resource management and autoscaling so that compute follows demand as a run’s needs shift.
• Build observability that makes it possible to understand what a run is doing and why it slowed down, stalled, or produced unexpected results.
• Build automation that detects and remediates common problems, and design interfaces that let engineers and automated tools operate runs safely.
• Work with researchers and performance engineers to make sure systems changes preserve training correctness and don’t introduce subtle nondeterminism.
• Remove classes of failure at their source through incident review, testing, and redesign, and write clear design documents for what you build.

Skills

Python

Systems languages

Distributed systems

Performance debugging

Communication writing

Throughput/latency analysis

Education

Bachelor's degree

Tools

Kubernetes

Sandboxed execution

Job description

Anthropic in San Francisco is seeking a Research Engineer to join the Distributed Systems team within RL Engineering. You will work on scheduling, data movement, autoscaling, and observability for large RL workloads, designing robust, scalable infrastructure.

The role emphasizes principled problem solving, cross-functional collaboration with researchers and performance engineers, and building systems that keep progressing despite hardware and network challenges.