Research Scientist / Engineer – Training Infrastructure

lumaai

United States

Remote

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

lumaai is seeking an engineer to design and optimize distributed training systems across thousands of GPUs. You will contribute to the research and implementation of advanced parallelism strategies for large multimodal foundation models, ensuring high throughput and stability in sprawling GPU fleets.

The role requires deep GPU cluster knowledge, experience with PyTorch-based distributed training, and familiarity with NCCL/MPI.

Qualifications

  • Extensive hands-on experience with distributed PyTorch training and the parallelisms used in foundation-model development.
  • Deep understanding of GPU cluster architecture, including networking and storage subsystems.
  • Familiarity with communication libraries such as NCCL and MPI, paired with a track record of distributed-system optimization.

Responsibilities

  • Design, implement, and optimize distributed training systems that scale to thousands of GPUs while preserving throughput and stability
  • Research and integrate advanced parallelization strategies, including FSDP, tensor, pipeline, and expert parallelism
  • Build monitoring, visualization, and debugging tooling that makes large training runs observable and diagnosable end to end
  • Tune training stability, convergence behavior, and resource utilization across massive GPU fleets

Skills

Distributed PyTorch training
GPU cluster architecture
NCCL
MPI
Parallelization strategies
Training stability optimization

Tools

Containerization
Orchestration
Cloud infrastructure

Job description

Role overview

This position sits at the intersection of research and infrastructure engineering, focused on building the distributed training systems that power large-scale multimodal foundation models running across thousands of GPUs. The work spans hard PyTorch and CUDA engineering alongside distributed-systems design, with an emphasis on advanced parallelism, training stability, and high utilization across massive clusters. It is built for engineers who have already solved real foundation-model training problems rather than those still ramping into distributed systems.

Responsibilities
  • Design, implement, and optimize distributed training systems that scale to thousands of GPUs while preserving throughput and stability
  • Research and integrate advanced parallelization strategies, including FSDP, tensor, pipeline, and expert parallelism
  • Build monitoring, visualization, and debugging tooling that makes large training runs observable and diagnosable end to end
  • Tune training stability, convergence behavior, and resource utilization across massive GPU fleets
First 90 days
  • Days 1–30 — Immerse and diagnose the current training stack, identifying where stability and utilization break down at scale
  • Days 30–60 — Ship a parallelization or stability improvement that measurably helps a real production training run
  • Days 60–90 — Build the monitoring and reliability tooling that keeps multi-thousand-GPU runs healthy and efficient
Requirements
  • Extensive hands-on experience with distributed PyTorch training and the parallelisms used in foundation-model development
  • Deep understanding of GPU cluster architecture, including networking and storage subsystems
  • Familiarity with communication libraries such as NCCL and MPI, paired with a track record of distributed-system optimization
Nice to have
  • Strong Linux systems administration and scripting skills
  • Experience managing training runs across 100+ GPU deployments
  • Background with containerization, orchestration, and cloud infrastructure
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Research Scientist / Engineer – Training Infrastructure
Research Scientist / Engineer – Training Infrastructure

Luma • Redwood City (CA)

On-site
USD 210,000 - 260,000
Copy of Research Scientist / Engineer – Training Infrastructure
Copy of Research Scientist / Engineer – Training Infrastructure

Luma AI • United States

Remote
USD 180,000 - 240,000
Research Scientist / Engineer – Training Infrastructure
Research Scientist / Engineer – Training Infrastructure

Luma • United States

On-site
USD 195,000 - 395,000
Research Scientist / Engineer – Training Infrastructure
Research Scientist / Engineer – Training Infrastructure

Luma • Seattle (WA)

On-site
USD 195,000 - 395,000
Research Scientist / Engineer – Training Infrastructure
Research Scientist / Engineer – Training Infrastructure

Luma • New York (NY)

On-site
USD 195,000 - 395,000
Training / AI Infrastructure
Training / AI Infrastructure

Genesis • San Carlos (CA)

On-site
USD 180,000 - 240,000
Training / AI Infrastructure
Training / AI Infrastructure

Genesis AI • United States

Hybrid
USD 190,000 - 240,000
Training / AI Infrastructure
Training / AI Infrastructure

Genesis AI • San Carlos (CA)

On-site
USD 180,000 - 240,000
Training / AI Infrastructure
Training / AI Infrastructure

Genesis AI • San Francisco (CA)

On-site
USD 180,000 - 260,000
Member of Technical Staff — Training Infrastructure
Member of Technical Staff — Training Infrastructure

Kindredventures • San Francisco (CA)

On-site
USD 180,000 - 240,000