Distributed Training Engineer for Multi-GPU Models

Openkrill; United States; Remote; USD 180,000 - 260,000; Full time

Job summary

Luma is building distributed training systems to power large-scale multimodal models that run across thousands of GPUs, enabling researchers to focus on innovation atop reliable, efficient infrastructure.

We tackle hard PyTorch, CUDA, and distributed-systems challenges—highly parallelized, stable, and scalable training across massive clusters. If you’ve not worked at this depth with FSDP and multi-node training, this is not the role for you.

Qualifications

• Extensive distributed PyTorch training and parallelism in foundation-model training.
• Deep understanding of GPU clusters, networking, and storage systems.
• Familiarity with communication libraries (NCCL, MPI) and distributed-system optimization.
• Strong Linux systems administration and scripting.

Responsibilities

• Design, implement, and optimize distributed training systems for models across thousands of GPUs.
• Research and implement advanced parallelization (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel).
• Build monitoring, visualization, and debugging tools for large-scale training runs.
• Optimize training stability, convergence, and resource utilization across massive clusters.

Skills

Distributed PyTorch training

GPU clusters

NCCL/MPI

Parallelism

CUDA

Tools

Containerization

Cloud infrastructure

Job description

Luma is building distributed training systems to power large-scale multimodal models that run across thousands of GPUs, enabling researchers to focus on innovation atop reliable, efficient infrastructure.

We tackle hard PyTorch, CUDA, and distributed-systems challenges—highly parallelized, stable, and scalable training across massive clusters. If you’ve not worked at this depth with FSDP and multi-node training, this is not the role for you.