Machine Learning Engineer — Reinforcement Learning

Mohamed bin Zayed University of Artificial Intelligence; Sunnyvale (CA); On-site; USD 150,000 - 450,000; Full time

Benefits offered by this job

Comprehensive benefits

Discretionary performance bonus

Paid time off

Job summary

Mohamed bin Zayed University of Artificial Intelligence seeks an RL infrastructure engineer to extend and scale end-to-end RL training systems alongside world-class researchers and engineers. You will work to extend distributed training frameworks, integrate rollout generation and policy updates, and build robust multi-node launch and monitoring tools.

You will own experiment tracking, metrics logging, and job monitoring, ensuring reliability and maintainability of ML infrastructure at scale.

Qualifications

• 5+ years of experience in ML systems, infra, or distributed training.
• Experience modifying distributed ML frameworks (e.g., DeepSpeed, FSDP, FairScale, Horovod).
• Strong software engineering fundamentals (Python, systems design, testing).
• Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) and debugging skills (e.g., NCCL/GLOO).
• Hands-on experience developing an LLM RL pipeline, with ownership across rollout/inference and distributed training integration.

Responsibilities

• Extend distributed training frameworks to support new use cases.
• Integrate rollout generation, reward computation, trajectory processing, and policy updates.
• Build robust config and launch systems across multi-node, multi-GPU clusters.
• Own experiment tracking, metrics logging, and job monitoring for external visibility.
• Improve training system reliability, maintainability, and performance.

Skills

ML systems

Distributed training

Python

Multi-node debugging

Education

Bachelor's/Master's in CS/ML

Tools

DeepSpeed

FSDP

FairScale

Horovod

Slurm

Kubernetes

Ray

NCCL

Job description

About the Institute of Foundation Models

We are a dedicated research lab for building, understanding, using,and risk-managing foundation models. Our mandate is to advanceresearch, nurture the next generation of AI builders, and drivetransformative contributions to a knowledge-driven economy.

As part of our team, you’ll have the opportunity to work on thecore of cutting-edge foundation model training, alongsideworld-class researchers, data scientists, and engineers, tacklingthe most fundamental and impactful challenges in AIdevelopment.

You will participate in the development ofgroundbreaking AI solutions that have the potential to reshapeentire industries. Strategic and innovative problem-solving skillswill be instrumental in establishing MBZUAI as a global hub forhigh-performance computing in deep learning, driving impactfuldiscoveries that inspire the next generation of AI pioneers.

The Role

We’re looking for an RL infrastructure engineer to help extend andscale our end-to-end RL training systems. You’ll work side-by-sidewith world-class researchers and engineers to:

• Extend distributed training frameworks (e.g., DeepSpeed, FSDP,FairScale, Horovod)
• Integrate rollout generation, reward computation, trajectoryprocessing, policy updates, and weight synchronization
• Build robust config + launch systems across multi-node,multi-GPU clusters
• Own experiment tracking, metrics logging, and job monitoringfor external visibility
• Improve training system reliability, maintainability, andperformance

Hands-on experience developing end-to-end RL infrastructure isrequired. Strong infrastructure and systems experience is what wevalue most.

Key Responsibilities

• Distributed Framework Ownership – Extend or modify trainingframeworks (e.g., DeepSpeed, FSDP) to support new use cases andarchitectures.
• Training & Inference Integration – Connect training workerswith rollout generation engines (e.g., vLLM, SGLang), includingtrajectory exchange and policy-weight synchronization.
• Launch Config & Debugging – Create and debug multi-nodelaunch scripts with flexible batch sizes, parallelism strategies,and hardware targets.
• Metrics & Monitoring – Build systems for experimenttracking, job monitoring, and logging usable by collaborators andresearchers.
• Infra Engineering – Write production-quality code and tests forML infra in PyTorch or JAX; ensure reliability and maintainabilityat scale.
• RL Pipeline Development – Implement reward/verifier integrationand trajectory processing, and validate log probabilities, tokenmasks, and loss inputs with researchers.
• RL Execution & Recovery – Coordinate rollout and trainingworkers, including checkpoint/restart and failure recovery; trackpolicy versions and sample staleness when using asynchronousexecution.

Qualifications

Must-Haves:

• 5+ years of experience in ML systems, infra, or distributedtraining
• Experience modifying distributed ML frameworks (e.g.,DeepSpeed, FSDP, FairScale, Horovod)
• Strong software engineering fundamentals (Python, systemsdesign, testing)
• Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) anddebugging skills (e.g., NCCL/GLOO)
• Ability to implement algorithms across GPUs/nodes based onmathematical specs
• Experience working on an ML platform/ infrastructure, and/ordistributed inference optimization team
• Experience with large-scale machine learning workloads (strongML fundamentals)
• Hands-on experience developing an LLM RL pipeline, withownership across rollout/inference and distributed trainingintegration.
• Working knowledge of policy optimization methods such as PPO orGRPO, sufficient to implement and debug sampling, logprobabilities, and policy updates.

Nice-to-Haves:

• Exposure to mixed-precision training (e.g., bf16, fp8) withaccuracy validation
• Familiarity with performance profiling, kernel fusion, ormemory optimization
• Open-source contributions or published research (MLSys, ICML,NeurIPS)
• CUDA or Triton kernel experience
• Experience with large-scale pre-training
• Experience building custom training pipelines at scale andmodifying them for custom needs
• Deep familiarity with training infrastructure and performancetuning
• Experience with asynchronous RL, multi-turn or tool-usingrollout environments, or custom reward/verifier systems.

Compensation

$150,000 – $450,000/year + discretionary performance bonus +benefits

The posted salary range represents the company’s good faithestimate of the base compensation for this position upon hire. Theactual base compensation offered may vary within this rangedepending on individual qualifications, including but not limitedto relevant skills, experience, education, certifications, andspecific business needs. In addition to base salary, totalcompensation includes an annual discretionary performance bonus andcomprehensive benefits.

Benefits Include

• Comprehensive medical, dental, and vision benefits
• Bonus
• 401K Plan
• Generous paid time off, sick leave and holidays
• Paid Parental Leave
• Employee Assistance Program
• Life insurance and disability

We may use artificial intelligence (AI) tools to support parts ofthe hiring process, such as reviewing applications, analyzingresumes, or assessing responses and identifying potentialinconsistencies or verification signals in application materialsbased on available information. These tools assist our recruitmentteam but do not replace human judgment. Final hiring decisions areultimately made by humans. If you would like more information abouthow your data is processed, please contact us.