Senior Distributed AI Training Architect

Luma AI; United States; Remote; USD 180,000 - 280,000; Full time

Job summary

Luma AI is seeking an engineer to design and optimize distributed training systems for large multimodal models. You will work on multi-GPU clusters, implement advanced parallelization, and build tools to monitor and debug massive training runs.

Requires extensive distributed PyTorch experience, deep GPU cluster knowledge, and familiarity with NCCL and MPI. Experience with containerization and orchestration is beneficial. This role is a high-depth engineering position at Luma AI.

Qualifications

• Extensive distributed PyTorch training across foundation-model scales.
• Deep understanding of GPU clusters, networking, and storage systems.
• Familiarity with NCCL and MPI for efficient communication.

Responsibilities

• Design, implement, and optimize distributed training systems for models across thousands of GPUs.
• Research and implement advanced parallelization (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel).
• Build monitoring, visualization, and debugging tools for large-scale training runs.

Skills

Distributed PyTorch

FSDP

Tensor Parallel

Pipeline Parallel

Expert Parallel

GPU clusters

Training stability

Resource tuning

Tools

NCCL

MPI

CUDA

Docker

Kubernetes

Job description

Luma AI is seeking an engineer to design and optimize distributed training systems for large multimodal models. You will work on multi-GPU clusters, implement advanced parallelization, and build tools to monitor and debug massive training runs.

Requires extensive distributed PyTorch experience, deep GPU cluster knowledge, and familiarity with NCCL and MPI. Experience with containerization and orchestration is beneficial. This role is a high-depth engineering position at Luma AI.