Senior RL Infrastructure Engineer | Scalable Training

Mohamed bin Zayed University of Artificial Intelligence; Sunnyvale (CA); On-site; USD 150,000 - 450,000; Full time

Benefits offered by this job

Comprehensive benefits

Discretionary performance bonus

Paid time off

Job summary

Mohamed bin Zayed University of Artificial Intelligence seeks an RL infrastructure engineer to extend and scale end-to-end RL training systems alongside world-class researchers and engineers. You will work to extend distributed training frameworks, integrate rollout generation and policy updates, and build robust multi-node launch and monitoring tools.

You will own experiment tracking, metrics logging, and job monitoring, ensuring reliability and maintainability of ML infrastructure at scale.

Qualifications

• 5+ years of experience in ML systems, infra, or distributed training.
• Experience modifying distributed ML frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod).
• Strong software engineering fundamentals (Python, systems design, testing).
• Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and debugging skills (e.g., NCCL/GLOO).
• Hands-on experience developing an LLM RL pipeline, with ownership across rollout/inference and distributed training integration.

Responsibilities

• Extend distributed training frameworks to support new use cases.
• Integrate rollout generation, reward computation, trajectory processing, and policy updates.
• Build robust config and launch systems across multi-node, multi-GPU clusters.
• Own experiment tracking, metrics logging, and job monitoring for external visibility.
• Improve training system reliability, maintainability, and performance.

Skills

ML systems

Distributed training

Python

Multi-node debugging

Education

Bachelor's/Master's in CS/ML

Tools

DeepSpeed

FSDP

FairScale

Horovod

Slurm

Kubernetes

Ray

NCCL

Job description

Mohamed bin Zayed University of Artificial Intelligence seeks an RL infrastructure engineer to extend and scale end-to-end RL training systems alongside world-class researchers and engineers. You will work to extend distributed training frameworks, integrate rollout generation and policy updates, and build robust multi-node launch and monitoring tools.

You will own experiment tracking, metrics logging, and job monitoring, ensuring reliability and maintainability of ML infrastructure at scale.