Senior Deep Learning Infra Architect for Large-Scale Training

Jobtailor; California (MO); On-site; USD 180,000 - 280,000; Full time

Job summary

Jobtailor is seeking an experienced senior infrastructure engineer to design, scale, and harden deep learning training systems across large GPU clusters. You will optimize the training stack, data pipelines, and orchestration layers while collaborating with ML researchers and platform teams to boost efficiency and availability.

The role emphasizes building robust libraries for massive datasets and ensuring scalability with growing GPU capacity, all while keeping developer productivity high.

Qualifications

• BS, MS, or PhD in Computer Science, Electrical/Computer Engineering, or a related field, or equivalent experience.
• 12+ years of professional experience building and scaling high‑performance distributed systems, ideally in ML, HPC, or large‑scale data infrastructure.
• Extensive knowledge in deep learning frameworks (PyTorch is preferred), large‑scale training (DDP/FSDP, NCCL, tensor/pipeline parallelism), and performance profiling.
• Strong systems background: datacenter networking (RoCE, IB), parallel filesystems (Lustre), storage systems, schedulers (Slurm, Kubernetes, etc.).
• Proficiency in Python and C++, with experience writing production‑grade libraries, orchestration layers, and automation tools.
• Ability to work closely with multi‑functional teams (ML researchers, infra engineers, product leads) and translate requirements into robust systems.

Responsibilities

• Crafting, scaling, and hardening deep learning infrastructure libraries and frameworks for training on multi‑thousand GPU clusters.
• Improving efficiency throughout the training stack: data loaders, distributed training, scheduling, and performance monitoring.
• Building robust training pipelines and libraries to handle massive video datasets and enable rapid experimentation.
• Collaborating with researchers, model engineers, and internal platform teams to enhance efficiency, minimize stalls, and improve training availability.
• Owning core infrastructure components such as orchestration libraries, distributed training frameworks, and fault‑resilient training systems.
• Partnering with leadership to ensure infrastructure scales with growing GPU capacity and dataset size while maintaining developer efficiency and stability.

Skills

Distributed systems

Python

C++

Deep learning

Collaboration

Performance profiling

Education

PhD in Computer Science or related field

MS in Computer Science or related field

Tools

PyTorch

DDP/FSDP

NCCL

Slurm

Kubernetes

Lustre

RoCE

Job description

Jobtailor is seeking an experienced senior infrastructure engineer to design, scale, and harden deep learning training systems across large GPU clusters. You will optimize the training stack, data pipelines, and orchestration layers while collaborating with ML researchers and platform teams to boost efficiency and availability.

The role emphasizes building robust libraries for massive datasets and ensuring scalability with growing GPU capacity, all while keeping developer productivity high.