LLM Pre-training & Distributed Engineer (AI Infrastructure)

Hyphen Connect Limited; San Francisco (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

Hyphen Connect Limited is seeking a highly skilled LLM Pre-training & Distributed Systems Engineer to optimize large-scale machine learning training runs. The ideal candidate will have deep expertise in GPU clusters and extensive systems engineering experience to ensure efficiency and reliability in training processes.

Key responsibilities include orchestrating distributed training runs, optimizing networking and memory management, and automating checkpointing during long training periods. If you have a passion for cutting-edge technology and system architecture, we want to hear from you.

Qualifications

• Deep expertise in orchestrating distributed training runs across multiple GPUs.
• Strong systems engineering skills with experience in C++, CUDA, and Python.

Responsibilities

• Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
• Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
• Automate checkpointing and failure recovery during month-long training runs.

Skills

3D parallelism expertise

Systems engineering (C++, CUDA, Python)

Job description

We are seeking a highly skilled LLM Pre-training & Distributed Systems Engineer. This role is essential for orchestrating large-scale machine learning training runs and optimizing distributed infrastructure. The ideal candidate will have a deep understanding of GPU clusters and extensive experience in system engineering to ensure efficient and reliable training processes.

Responsibilities

• Orchestrate distributed training runs across 1,000+ GPUs using PyTorch, DeepSpeed, or Megatron-LM.
• Optimize networking (InfiniBand/RDMA) and memory management to prevent out-of-memory errors.
• Automate checkpointing and failure recovery during month-long training runs.

Required Skills

• Deep expertise in 3D parallelism (Data, Tensor, Pipeline).
• Strong systems engineering background (C++, CUDA, Python).