Senior Distributed Systems Engineer — High-Perf GPU

Lever, Inc.; Sunnyvale (CA); On-site; USD 180,000 - 250,000; Full time

Job summary

The Institute of Foundation Models designs ultra-scale GPU systems to train foundation models and optimize cross-layer performance and scalability. We seek a deeply technical engineer to co-design and optimize the communication stack for large-scale distributed training, including MoE workloads, hybrid parallelism, and fault-tolerant execution across thousands of GPUs.

This role emphasizes performance engineering, distributed debugging, and production-grade optimizations at the boundary of

Qualifications

• Experience optimizing distributed training at 1,000+ GPU scale.
• Hands-on with RDMA, InfiniBand, RoCE, and GPUDirect RDMA.
• Deep familiarity with NCCL and/or UCX internals.
• Systems programming in C/C++, Rust, or Go.
• Experience with PyTorch in large-scale training.
• Ability to profile and fix communication bottlenecks.

Responsibilities

• Design and optimize expert-parallel and hybrid-parallel communication patterns.
• Drive high-performance hierarchical collectives for MoE workloads.
• Co-design runtime orchestration with communication topology awareness.
• Reduce tail latency and improve determinism across thousands of GPUs.
• Architect fault-tolerant distributed execution under real-world cluster failures.

Skills

RDMA/InfiniBand optimization

NCCL/UCX internals

C/C++ / Rust / Go systems programming

PyTorch large-scale training

Distributed debugging

Troubleshooting communication bottlene

Tools

NCCL

UCX

GPUDirect RDMA

InfiniBand

RoCE

Job description

The Institute of Foundation Models designs ultra-scale GPU systems to train foundation models and optimize cross-layer performance and scalability. We seek a deeply technical engineer to co-design and optimize the communication stack for large-scale distributed training, including MoE workloads, hybrid parallelism, and fault-tolerant execution across thousands of GPUs.

This role emphasizes performance engineering, distributed debugging, and production-grade optimizations at the boundary of