Distributed AI Training Engineer (GPU Clusters)

lumalabs-ai; San Francisco (CA); On-site; USD 188,000 - 395,000; Full time

Job summary

Luma AI is seeking engineers for the Training Infrastructure team to design and maintain distributed systems that enable training of large-scale multimodal models on thousands of GPUs. You will work with researchers to push the boundaries of AI model development.

Ideal candidates have extensive PyTorch distributed training experience and a deep understanding of GPU clusters, networking, and storage, plus familiarity with NCCL/MPI and Linux scripting.

Qualifications

• Extensive experience with distributed PyTorch training and model-scale parallelism.
• Deep understanding of GPU clusters, networking, and storage systems.
• Familiarity with communication libraries (NCCL, MPI) and distributed system optimization.
• Preferred: strong Linux systems administration and scripting capabilities.
• Preferred: experience managing training runs across >100 GPUs.
• Preferred: experience with containerization, orchestration, and cloud infrastructure.

Responsibilities

• Design, implement, and optimize efficient distributed training systems for models with thousands of GPUs
• Research and implement advanced parallelization techniques (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel)
• Build monitoring, visualization, and debugging tools for large-scale training runs
• Optimize training stability, convergence, and resource utilization across massive clusters

Skills

Distributed PyTorch

GPUs & clusters

NCCL/MPI

Linux scripting

Containerization

Cloud infrastructure

Job description

Luma AI is seeking engineers for the Training Infrastructure team to design and maintain distributed systems that enable training of large-scale multimodal models on thousands of GPUs. You will work with researchers to push the boundaries of AI model development.

Ideal candidates have extensive PyTorch distributed training experience and a deep understanding of GPU clusters, networking, and storage, plus familiarity with NCCL/MPI and Linux scripting.