Senior Distributed Systems Engineer MoE at Scale

Ifm Us; Sunnyvale (CA); On-site; USD 190,000 - 260,000; Full time

Benefits offered by this job

Bonus

401K Plan

Generous paid time off

Job summary

The Institute of Foundation Models (IFM) is seeking a deeply technical engineer to co‑design and optimize the communication stack for large‑scale distributed training, including hybrid parallelism and MoE workloads.

This role focuses on performance engineering, distributed debugging, and cross‑layer optimization across thousands of GPUs, with a strong emphasis on fault‑tolerant execution and topology‑aware design.

Qualifications

• Experience optimizing distributed training at large GPU scales.
• Hands‑on with RDMA, InfiniBand, RoCE, and GPUDirect RDMA.
• Familiarity with NCCL and UCX internals.
• Strong systems programming ability in C/C++, Rust, or Go.

Responsibilities

• Design and optimize communication for large‑scale distributed training (thousand+ GPUs).
• Drive high‑performance hierarchical collectives for MoE workloads.
• Co‑design runtime orchestration with topology awareness.
• Debug and profile training performance bottlenecks in communication.

Skills

NCCL/UCX internals

GPUDirect RDMA

RDMA/InfiniBand/RoCE

C/C++/Rust/Go

PyTorch familiarity

distributed debugging

performance engineering

Education

Master’s degree or Bachelor’s + 1 year experience

Tools

NCCL

UCX

InfiniBand

Job description

The Institute of Foundation Models (IFM) is seeking a deeply technical engineer to co‑design and optimize the communication stack for large‑scale distributed training, including hybrid parallelism and MoE workloads.

This role focuses on performance engineering, distributed debugging, and cross‑layer optimization across thousands of GPUs, with a strong emphasis on fault‑tolerant execution and topology‑aware design.