Scale-Up AI Training Systems Engineer

Dover; Paris; On-site; EUR 120,000 - 180,000; Full time

Job summary

UMA is seeking an AI Infra/Engineer — Large-scale Training to build scalable, reliable systems for humanoid robot learning. You’ll turn experimental training code into robust pipelines across thousands of GPUs, enabling rapid iteration with researchers.

You will own distributed execution, optimize memory and communication, and ensure long-running jobs run deterministically. You’ll contribute to pretraining, mid-training, and post-training, shaping the foundation of future models.

Qualifications

• Experience building ML training infrastructure and distributed systems.
• Strong Python and PyTorch with DDP/FSDP or Megatron knowledge.
• Experience with multiprocessing, asynchronous execution, and GPU memory management.
• Understanding losses, optimizers, gradients, and numerical precision.
• Experience profiling accelerator workloads with PyTorch Profiler or Nsight Systems.
• Pragmatic engineering mindset to turn research into reliable systems.
• Clear communication and collaboration with researchers.
• Bonus: RL infra, distributed rollouts, replay buffers, actor–learner.
• Bonus: C++, CUDA, Triton, custom kernels.
• Bonus: Slurm or Kubernetes, InfiniBand/RoCE, storage caching.
• Bonus: open-source training tools or technical writing.

Responsibilities

• Build distributed pretraining systems across GPUs and nodes.
• Develop RL and post-training execution infrastructure.
• Maximize MFU and eliminate bottlenecks in the training loop.
• Integrate accelerated kernels and execution techniques.
• Build training-time data movement and buffering.
• Turn research prototypes into maintainable training systems.
• Diagnose distributed and numerical failures.
• Deep numerical debugging and run determinism.
• Build reliable checkpointing and fault-tolerant resumption.
• Strengthen observability and reproducibility.

Skills

Python

PyTorch

Distributed training

Profiling tools

GPU memory management

DDP / FSDP

Megatron

C++

CUDA

Kubernetes

Slurm

Nsight Systems

Tools

PyTorch Profiler

Nsight Systems

TorchScript

Job description

UMA is seeking an AI Infra/Engineer — Large-scale Training to build scalable, reliable systems for humanoid robot learning. You’ll turn experimental training code into robust pipelines across thousands of GPUs, enabling rapid iteration with researchers.

You will own distributed execution, optimize memory and communication, and ensure long-running jobs run deterministically. You’ll contribute to pretraining, mid-training, and post-training, shaping the foundation of future models.