Distributed RL Systems Engineer — Flexible Hours & Scale

Anthropic Limited; Seattle (WA); On-site; USD 500,000 - 850,000; Full time

Benefits offered by this job

Generous vacation

Parental leave

Flexible working hours

Lovely office space

Job summary

Anthropic Limited is seeking a Research Engineer to join the Distributed Systems team within RL Engineering. You will work across training, sampling, and environment execution to keep a large-scale RL system progressing as hardware and workloads shift.

You will build fault tolerance, autoscaling, and observability to support ongoing, reliable AI research initiatives. Responsibilities emphasize cross-layer reasoning, design documents, and collaboration with researchers and performance engineers

Qualifications

• Strong software engineering skills in Python and at least one systems language.
• Experience designing, building, and operating large-scale distributed systems in production.
• Deep understanding of distributed systems fundamentals including consistency and recovery.
• Ability to reason about throughput, latency, and resource costs across compute, memory, storage, and network.
• Experience debugging complex failures across many hosts and services.

Responsibilities

• Design, build, and operate the distributed systems that run RL at scale across training, sampling, and environment execution.
• Identify and remove current system bottlenecks, including scheduling, data movement, storage, networking, or coordination.
• Build fault tolerance into every layer with failure detection, isolation, and recovery for long-running jobs.
• Design resource management and autoscaling to match compute needs with demand.
• Develop observability to understand run behavior and performance.
• Create automation to detect and remediate problems and enable safe operation of runs.
• Collaborate with researchers and engineers to preserve training correctness and reduce nondeterminism.
• Document design decisions and rationales to guide future work.

Skills

Python

Systems language

Distributed systems

Performance analysis

Debugging

Technical writing

Education

Bachelor's degree

Field relevant to role

Tools

Kubernetes

Job description

Anthropic Limited is seeking a Research Engineer to join the Distributed Systems team within RL Engineering. You will work across training, sampling, and environment execution to keep a large-scale RL system progressing as hardware and workloads shift.

You will build fault tolerance, autoscaling, and observability to support ongoing, reliable AI research initiatives. Responsibilities emphasize cross-layer reasoning, design documents, and collaboration with researchers and performance engineers