Distributed ML Infra Engineer for Large-Scale Pre-Training

Mohamed bin Zayed University of Artificial Intelligence

Sunnyvale (CA)

On-site

USD 150,000 - 450,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Medical, dental, vision
Bonus
401K Plan
Paid time off
Parental Leave
Employee Assistance Program
Life insurance

Job summary

MBZUAI seeks a distributed ML infrastructure engineer to extend and scale our training systems, working alongside world-class researchers to push the frontier of foundation models. You will help implement distributed optimizers, build multi-node launch scripts, and ensure reliability across GPU clusters.

This role emphasizes experimentation, performance tuning, and robust data loading and checkpointing, with a strong focus on PyTorch or JAX.

Qualifications

  • 5+ years in ML systems, infra, or distributed training.
  • Experience modifying distributed ML frameworks.
  • Strong Python software fundamentals.
  • Multi-node experience with Slurm, Kubernetes, Ray.
  • Ability to implement algorithms across GPUs/nodes based on math specs.

Responsibilities

  • Extend or modify training frameworks for new use cases.
  • Translate optimizer specs into distributed implementations.
  • Create and debug multi-node launch scripts with flexible batch sizes and parallelism strategies.
  • Select and validate data, tensor, pipeline, expert, and context parallelism strategies.
  • Build systems for experiment tracking, metrics logging, and job monitoring.
  • Write production-quality ML infra code and tests in PyTorch or JAX.
  • Build and maintain data-loading and checkpoint/restart workflows.

Skills

ML systems
Distributed training
Python
Multi-node debugging
Slurm
Kubernetes
Ray
NCCL/GLOO
GPU parallelism

Education

Bachelor's degree

Tools

DeepSpeed
FSDP
FairScale
Horovod
PyTorch
JAX

Job description

MBZUAI seeks a distributed ML infrastructure engineer to extend and scale our training systems, working alongside world-class researchers to push the frontier of foundation models. You will help implement distributed optimizers, build multi-node launch scripts, and ensure reliability across GPU clusters.

This role emphasizes experimentation, performance tuning, and robust data loading and checkpointing, with a strong focus on PyTorch or JAX.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Distributed ML Infra Engineer - Large-Scale Pre-Training
Distributed ML Infra Engineer - Large-Scale Pre-Training

Institute of Foundation Models • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, and vision benefits
Bonus
401K Plan
+4
Distributed Training Infra Engineer for Large Models
Distributed Training Infra Engineer for Large Models

Kindredventures • San Francisco (CA)

On-site
USD 180,000 - 240,000
Distributed ML Training Infra Engineer
Distributed ML Training Infra Engineer

Thinking Machines Lab • San Francisco (CA)

On-site
USD 300,000 - 400,000
Health insurance
Dental insurance
Vision benefits
+3
Senior ML Infra Research Engineer — Distributed Training
Senior ML Infra Research Engineer — Distributed Training

present of AI • San Francisco (CA)

On-site
USD 180,000 - 250,000
Distributed ML Training Engineer - Large Models
Distributed ML Training Engineer - Large Models

Reflection AI • San Francisco (CA)

On-site
USD 180,000 - 300,000
Top-tier compensation
Stock options
Health & wellness
+2
Senior Research Engineer, Large-Scale ML Pre-training
Senior Research Engineer, Large-Scale ML Pre-training

Magic • San Francisco (CA)

On-site
USD 275,000 - 474,000
Equity in total compensation
401(k) with matching
Health, dental and vision insurance
+1
Machine Learning Engineer — Pre-training (LLM)
Machine Learning Engineer — Pre-training (LLM)

Institute of Foundation Models • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, and vision benefits
Bonus
401K Plan
+4
ML Infra Engineer - Scale GPU Clusters & Simulation
ML Infra Engineer - Scale GPU Clusters & Simulation

Zeromatter • Mountain View (CA)

On-site
USD 100,000 - 300,000
Equity
Medical, dental, vision insurance
ML Infra Engineer: Scale & Optimize Large-Scale Training
ML Infra Engineer: Scale & Optimize Large-Scale Training

Physical Intelligence • San Francisco (CA)

On-site
USD 180,000 - 240,000
Staff Engineer - Distributed AI Infrastructure & Systems
Staff Engineer - Distributed AI Infrastructure & Systems

Micro1 • United States

Remote
USD 180,000 - 260,000