Distributed RL Systems Engineer — Flexible Hours

Anthropic Limited; New York (NY); Hybrid; USD 500,000 - 850,000; Full time

Job summary

Anthropic Limited seeks a Research Engineer for the Distributed Systems team within RL Engineering. You will work on RL training environments, scheduling, data movement, and fault tolerance across a large-scale distributed system.

We value generalists who can move across layers, reason from first principles, and select impactful problems over those closest to prior experience. Expect a fast-evolving research setting with emphasis on correctness.

Qualifications

• Proficient in Python and at least one systems language (Rust, C++, or Go).
• Experience designing, building, and operating large-scale distributed systems in production.
• Strong understanding of distributed systems: consistency, coordination, consensus, and recovery.
• Ability to reason quantitatively about throughput, latency, and resource costs.
• Experience debugging complex failures across many hosts and services.
• Strong written communication, including design documents and incident writeups.

Responsibilities

• Design, build, and operate distributed systems for RL at scale (training, sampling, env execution).
• Identify and remove current system limits (scheduling, data movement, storage, networking, coordination).
• Impart fault tolerance across layers: detection, isolation, recovery for long-running jobs.
• Design resource management and autoscaling to follow demand.
• Develop observability to understand run behavior and performance issues.
• Create automation and safe interfaces for engineers and automated tools.
• Collaborate with researchers and performance engineers to preserve training correctness.
• Provide clear design documents and contribute to incident reviews.

Skills

Python

Rust/C++/Go

Distributed systems

Quantitative reasoning

Debugging complex failures

Technical writing

Education

Bachelor's degree or equivalent

Tools

Kubernetes

Job description

Anthropic Limited seeks a Research Engineer for the Distributed Systems team within RL Engineering. You will work on RL training environments, scheduling, data movement, and fault tolerance across a large-scale distributed system.

We value generalists who can move across layers, reason from first principles, and select impactful problems over those closest to prior experience. Expect a fast-evolving research setting with emphasis on correctness.