Machine Learning Engineer — Pre-training (LLM)
Mohamed bin Zayed University of Artificial Intelligence
Una candidatura completa in un minuto
Curriculum e lettera di presentazione personalizzati, pronti da inviare.
Vantaggi offerti da questo lavoro
Descrizione del lavoro
MBZUAI seeks a distributed ML infrastructure engineer to extend and scale our training systems, working alongside world-class researchers to push the frontier of foundation models. You will help implement distributed optimizers, build multi-node launch scripts, and ensure reliability across GPU clusters.
This role emphasizes experimentation, performance tuning, and robust data loading and checkpointing, with a strong focus on PyTorch or JAX.
Competenze
- 5+ years in ML systems, infra, or distributed training.
- Experience modifying distributed ML frameworks.
- Strong Python software fundamentals.
- Multi-node experience with Slurm, Kubernetes, Ray.
- Ability to implement algorithms across GPUs/nodes based on math specs.
Mansioni
- Extend or modify training frameworks for new use cases.
- Translate optimizer specs into distributed implementations.
- Create and debug multi-node launch scripts with flexible batch sizes and parallelism strategies.
- Select and validate data, tensor, pipeline, expert, and context parallelism strategies.
- Build systems for experiment tracking, metrics logging, and job monitoring.
- Write production-quality ML infra code and tests in PyTorch or JAX.
- Build and maintain data-loading and checkpoint/restart workflows.
Conoscenze
Formazione
Strumenti
Descrizione del lavoro
We are a dedicated research lab for building, understanding, using,and risk-managing foundation models. Our mandate is to advanceresearch, nurture the next generation of AI builders, and drivetransformative contributions to a knowledge-driven economy.
As part of our team, you’ll have the opportunity to work on thecore of cutting-edge foundation model training, alongsideworld-class researchers, data scientists, and engineers, tacklingthe most fundamental and impactful challenges in AIdevelopment. You will participate in the development ofgroundbreaking AI solutions that have the potential to reshapeentire industries. Strategic and innovative problem-solving skillswill be instrumental in establishing MBZUAI as a global hub forhigh-performance computing in deep learning, driving impactfuldiscoveries that inspire the next generation of AI pioneers.
We’re looking for a distributed ML infrastructure engineer to helpextend and scale our training systems. You’ll work side-by-sidewith world-class researchers and engineers to:
- Extend distributed training frameworks (e.g., DeepSpeed, FSDP,FairScale, Horovod)
- Implement distributed optimizers from mathematical specs
- Build robust config + launch systems across multi-node,multi-GPU clusters
- Own experiment tracking, metrics logging, and job monitoringfor external visibility
- Improve training system reliability, maintainability, andperformance
Much of the work will support large-scale pre-training, andpre-training experience is required. Strong infrastructure andsystems experience is what we value most.
- Distributed Framework Ownership – Extend or modify trainingframeworks (e.g., DeepSpeed, FSDP) to support new use cases andarchitectures.
- Optimizer Implementation – Translate mathematical optimizerspecs into distributed implementations.
- Launch Config & Debugging – Create and debug multi-nodelaunch scripts with flexible batch sizes, parallelism strategies,and hardware targets. Select and validate data, tensor, pipeline,expert, and context parallelism strategies as appropriate for themodel and cluster.
- Metrics & Monitoring – Build systems for experimenttracking, job monitoring, and logging usable by collaborators andresearchers.
- Infra Engineering – Write production-quality code and tests forML infra in PyTorch or JAX; ensure reliability and maintainabilityat scale.
- Data Loading & Checkpointing – Build and maintaindata-loading and checkpoint/restart workflows, restoring model,optimizer, RNG, and data progress after interruptions.
- 5+ years of experience in ML systems, infra, or distributedtraining
- Experience modifying distributed ML frameworks (e.g.,DeepSpeed, FSDP, FairScale, Horovod)
- Strong software engineering fundamentals (Python, systemsdesign, testing)
- Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) anddebugging skills (e.g., NCCL/GLOO)
- Ability to implement algorithms across GPUs/nodes based onmathematical specs
- Experience working on an ML platform/ infrastructure, and/ordistributed inference optimization team
- Experience with large-scale machine learning workloads (strongML fundamentals)
- Experience with large-scale pre-training
- Exposure to mixed-precision training (e.g., bf16, fp8) withaccuracy validation
- Familiarity with performance profiling, kernel fusion, ormemory optimization
- Open-source contributions or published research (MLSys, ICML,NeurIPS)
- CUDA or Triton kernel experience
- Experience building custom training pipelines at scale andmodifying them for custom needs
- Deep familiarity with training infrastructure and performancetuning
$150,000 - $450,000 a year
$150,000 – $450,000/year + discretionary performance bonus +benefits
The posted salary range represents the company’s good faithestimate of the base compensation for this position upon hire. Theactual base compensation offered may vary within this rangedepending on individual qualifications, including but not limitedto relevant skills, experience, education, certifications, andspecific business needs. In addition to base salary, totalcompensation includes an annual discretionary performance bonus andcomprehensive benefits.
- Comprehensive medical, dental, and vision benefits
- Bonus
- 401K Plan
- Generous paid time off, sick leave and holidays
- Paid Parental Leave
- Employee Assistance Program
- Life insurance and disability
We may use artificial intelligence (AI) tools to support parts ofthe hiring process, such as reviewing applications, analyzingresumes, or assessing responses and identifying potentialinconsistencies or verification signals in application materialsbased on available information. These tools assist our recruitmentteam but do not replace human judgment. Final hiring decisions areultimately made by humans. If you would like more information abouthow your data is processed, please contact us.
We are a dedicated research lab for building, understanding, using,and risk-managing foundation models. Our mandate is to advanceresearch, nurture the next generation of AI builders, and drivetransformative contributions to a knowledge-driven economy.
As part of our team, you’ll have the opportunity to work on thecore of cutting-edge foundation model training, alongsideworld-class researchers, data scientists, and engineers, tacklingthe most fundamental and impactful challenges in AIdevelopment. You will participate in the development ofgroundbreaking AI solutions that have the potential to reshapeentire industries. Strategic and innovative problem-solving skillswill be instrumental in establishing MBZUAI as a global hub forhigh-performance computing in deep learning, driving impactfuldiscoveries that inspire the next generation of AI pioneers.
We’re looking for a distributed ML infrastructure engineer to helpextend and scale our training systems. You’ll work side-by-sidewith world-class researchers and engineers to:
- Extend distributed training frameworks (e.g., DeepSpeed, FSDP,FairScale, Horovod)
- Implement distributed optimizers from mathematical specs
- Build robust config + launch systems across multi-node,multi-GPU clusters
- Own experiment tracking, metrics logging, and job monitoringfor external visibility
- Improve training system reliability, maintainability, andperformance
Much of the work will support large-scale pre-training, andpre-training experience is required. Strong infrastructure andsystems experience is what we value most.
- Distributed Framework Ownership – Extend or modify trainingframeworks (e.g., DeepSpeed, FSDP) to support new use cases andarchitectures.
- Optimizer Implementation – Translate mathematical optimizerspecs into distributed implementations.
- Launch Config & Debugging – Create and debug multi-nodelaunch scripts with flexible batch sizes, parallelism strategies,and hardware targets. Select and validate data, tensor, pipeline,expert, and context parallelism strategies as appropriate for themodel and cluster.
- Metrics & Monitoring – Build systems for experimenttracking, job monitoring, and logging usable by collaborators andresearchers.
- Infra Engineering – Write production-quality code and tests forML infra in PyTorch or JAX; ensure reliability and maintainabilityat scale.
- Data Loading & Checkpointing – Build and maintaindata-loading and checkpoint/restart workflows, restoring model,optimizer, RNG, and data progress after interruptions.
- 5+ years of experience in ML systems, infra, or distributedtraining
- Experience modifying distributed ML frameworks (e.g.,DeepSpeed, FSDP, FairScale, Horovod)
- Strong software engineering fundamentals (Python, systemsdesign, testing)
- Proven multi-node experience (e.g., Slurm, Kubernetes, Ray) anddebugging skills (e.g., NCCL/GLOO)
- Ability to implement algorithms across GPUs/nodes based onmathematical specs
- Experience working on an ML platform/ infrastructure, and/ordistributed inference optimization team
- Experience with large-scale machine learning workloads (strongML fundamentals)
- Experience with large-scale pre-training
- Exposure to mixed-precision training (e.g., bf16, fp8) withaccuracy validation
- Familiarity with performance profiling, kernel fusion, ormemory optimization
- Open-source contributions or published research (MLSys, ICML,NeurIPS)
- CUDA or Triton kernel experience
- Experience building custom training pipelines at scale andmodifying them for custom needs
- Deep familiarity with training infrastructure and performancetuning
$150,000 - $450,000 a year
$150,000 – $450,000/year + discretionary performance bonus +benefits
The posted salary range represents the company’s good faithestimate of the base compensation for this position upon hire. Theactual base compensation offered may vary within this rangedepending on individual qualifications, including but not limitedto relevant skills, experience, education, certifications, andspecific business needs. In addition to base salary, totalcompensation includes an annual discretionary performance bonus andcomprehensive benefits.
- Comprehensive medical, dental, and vision benefits
- Bonus
- 401K Plan
- Generous paid time off, sick leave and holidays
- Paid Parental Leave
- Employee Assistance Program
- Life insurance and disability
We may use artificial intelligence (AI) tools to support parts ofthe hiring process, such as reviewing applications, analyzingresumes, or assessing responses and identifying potentialinconsistencies or verification signals in application materialsbased on available information. These tools assist our recruitmentteam but do not replace human judgment. Final hiring decisions areultimately made by humans. If you would like more information abouthow your data is processed, please contact us.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
