LLM Pre-training & Distributed Engineer (AI Infrastructure)

Hyphen Connect Limited; Oregon (WI); On-site; USD 100,000 - 130,000; Full time

Job summary

Hyphen Connect Limited is seeking a highly skilled LLM Pre-training & Distributed Systems Engineer to orchestrate large-scale machine learning training runs and optimize distributed infrastructure. Candidates should have a deep understanding of GPU clusters and extensive experience in systems engineering for efficient and reliable training processes.

This role involves optimizing networking and memory management during training and automating the recovery of checkpoints after failures. Interested candidates with strong expertise in parallelism, C++, CUDA, and Python are encouraged to apply.

Qualifications

• Deep expertise in 3D parallelism: Data, Tensor, Pipeline.
• Experience optimizing distributed systems for machine learning.
• Proficient in systems engineering with C++, CUDA, and Python.

Responsibilities

• Orchestrate distributed training runs across 1,000+ GPUs.
• Optimize networking and memory management to prevent errors.
• Automate checkpointing and failure recovery in training runs.

Skills

3D parallelism (Data, Tensor, Pipeline)

C++

CUDA

Python

Tools

PyTorch

DeepSpeed

Megatron-LM

Job description

We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.

Responsibilities

• Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
• Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
• Automate checkpointing and failure recovery during month-long training runs.

Required Skills

• Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
• Strong systems engineering background (C++, CUDA, Python).