LLM Distributed Training Engineer (1000+ GPUs)

Hyphen Connect Limited; San Francisco (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

Hyphen Connect Limited is seeking a highly skilled LLM Pre-training & Distributed Systems Engineer to optimize large-scale machine learning training runs. The ideal candidate will have deep expertise in GPU clusters and extensive systems engineering experience to ensure efficiency and reliability in training processes.

Key responsibilities include orchestrating distributed training runs, optimizing networking and memory management, and automating checkpointing during long training periods. If you have a passion for cutting-edge technology and system architecture, we want to hear from you.

Qualifications

• Deep expertise in orchestrating distributed training runs across multiple GPUs.
• Strong systems engineering skills with experience in C++, CUDA, and Python.

Responsibilities

• Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
• Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
• Automate checkpointing and failure recovery during month-long training runs.

Skills

3D parallelism expertise

Systems engineering (C++, CUDA, Python)

Job description

Hyphen Connect Limited is seeking a highly skilled LLM Pre-training & Distributed Systems Engineer to optimize large-scale machine learning training runs. The ideal candidate will have deep expertise in GPU clusters and extensive systems engineering experience to ensure efficiency and reliability in training processes.

Key responsibilities include orchestrating distributed training runs, optimizing networking and memory management, and automating checkpointing during long training periods. If you have a passion for cutting-edge technology and system architecture, we want to hear from you.