LLM Distributed Training Engineer (1000+ GPUs)

Hyphen Connect Limited; Oregon (WI); On-site; USD 100,000 - 130,000; Full time

Job summary

Hyphen Connect Limited is seeking a highly skilled LLM Pre-training & Distributed Systems Engineer to orchestrate large-scale machine learning training runs and optimize distributed infrastructure. Candidates should have a deep understanding of GPU clusters and extensive experience in systems engineering for efficient and reliable training processes.

This role involves optimizing networking and memory management during training and automating the recovery of checkpoints after failures. Interested candidates with strong expertise in parallelism, C++, CUDA, and Python are encouraged to apply.

Qualifications

• Deep expertise in 3D parallelism: Data, Tensor, Pipeline.
• Experience optimizing distributed systems for machine learning.
• Proficient in systems engineering with C++, CUDA, and Python.

Responsibilities

• Orchestrate distributed training runs across 1,000+ GPUs.
• Optimize networking and memory management to prevent errors.
• Automate checkpointing and failure recovery in training runs.

Skills

3D parallelism (Data, Tensor, Pipeline)

C++

CUDA

Python

Tools

PyTorch

DeepSpeed

Megatron-LM

Job description

Hyphen Connect Limited is seeking a highly skilled LLM Pre-training & Distributed Systems Engineer to orchestrate large-scale machine learning training runs and optimize distributed infrastructure. Candidates should have a deep understanding of GPU clusters and extensive experience in systems engineering for efficient and reliable training processes.

This role involves optimizing networking and memory management during training and automating the recovery of checkpoints after failures. Interested candidates with strong expertise in parallelism, C++, CUDA, and Python are encouraged to apply.