Copy of Research Scientist / Engineer – Training Infrastructure

Luma AI; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Luma AI seeks an engineer to design, implement, and optimize distributed training systems across thousands of GPUs. You’ll push reliability, efficiency, and scalability in real-world model training at scale.

You’ll lead advanced parallelization techniques (FSDP, tensor & pipeline parallelism) and build monitoring tools to keep massive runs observable and stable across clusters.

Qualifications

• Extensive distributed PyTorch training and parallelisms in foundation-model training.
• Deep understanding of GPU clusters, networking, and storage systems.
• Familiarity with communication libraries (NCCL, MPI) and distributed-system optimization.
• Strong Linux systems administration and scripting.
• Experience managing training runs across 100+ GPUs.
• Experience with containerization, orchestration, and cloud infrastructure.

Responsibilities

• Design and optimize distributed training systems across thousands of GPUs.
• Research and implement advanced parallelization (FSDP, Tensor, Pipeline, Expert).
• Build monitoring, visualization, and debugging tools for large-scale runs.
• Improve training stability, convergence, and resource utilization.
• First 90 Days: immerse, diagnose, ship, scale with measurable improvements.

Skills

Distributed PyTorch

FSDP

Tensor Parallel

Pipeline Parallel

Expert Parallel

NCCL

MPI

Linux Scripting

GPU Clusters

Cloud / Containers

Job description

You’ll build the distributed systems that train Luma’s large-scale multimodal models across thousands of GPUs, so researchers can focus on innovation on top of reliable, efficient, scalable infrastructure. This is hard PyTorch, CUDA, and distributed-systems work — advanced parallelism, training stability, and utilization across massive clusters. It fits an engineer who’s solved real problems training foundation models at scale. If you haven’t worked at the level of FSDP and multi-node training, this is the wrong depth.

What You’ll Own

• Design, implement, and optimize efficient distributed training systems for models across thousands of GPUs.
• Research and implement advanced parallelization (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel).
• Build monitoring, visualization, and debugging tools for large-scale training runs.
• Optimize training stability, convergence, and resource utilization across massive clusters.

First 90 Days

One way the first 90 could unfold.

• Days 1–30 — Immerse Diagnose: Learn the current training stack and where stability and utilization hurt at scale.
• Days 30–60 — Ship Validate: Land a parallelization or stability improvement that measurably helps a real training run.
• Days 60–90 — Scale Systemize: Build the monitoring and tooling that keeps large runs reliable and efficient.

What You Bring

• Extensive distributed PyTorch training and parallelisms in foundation-model training.
• Deep understanding of GPU clusters, networking, and storage systems.
• Familiarity with communication libraries (NCCL, MPI) and distributed-system optimization.
• Nice to HaveStrong Linux systems administration and scripting.
• Experience managing training runs across 100+ GPUs.
• Experience with containerization, orchestration, and cloud infrastructure.

About Luma:

Luma’s mission is to build unified general intelligence that can generate, understand, and operate in the physical world. We believe multimodality is critical for intelligence — the next step beyond language models comes from vision. Luma is an equal opportunity employer.