Senior GPU Performance Architect: Distributed Training

Jobtailor; Boston (MA); On-site; USD 180,000 - 240,000; Full time

Job summary

Jobtailor is seeking a senior capacity optimization lead to manage GPU and large-scale compute workloads across multi-node training environments. You will own the capacity model, including node counts, assumptions, and error bars, and defend capacity decisions with real financial impact.

You will optimize workload mix, preemptible fractions, and checkpointing economics, while improving training throughput and data locality across InfiniBand/RDMA networks and parallel file systems.

Qualifications

• Five or more years working with GPU and large-scale compute workloads.
• Real multi-node experience with distributed training performance and interconnects.
• System-level knowledge of GPU performance, including modern workloads.
• Deep understanding of AI training and inference performance.
• Understanding of step time, MFU, compute/communication overlap, and latency floors.

Responsibilities

• Own the capacity model, including node counts, assumptions and error bars.
• Defend capacity decisions when real money is committed.
• Own workload mix, preemptible fraction, and checkpointing economics over time.
• Explain performance results clearly to researchers in usable terms.
• Locate actual performance bottlenecks beyond profiler summaries.
• Measure and optimize parallel file systems, data locality, and interconnect behavior for multi-node GPU training.

Skills

GPU Workloads

Large-Scale Compute

Performance Bottleneck Analysis

AI Training and Inference

Data Locality Optimization

Capacity Modeling

Multi-Node GPU Training

Training Throughput

InfiniBand Networking

RDMA Networking

Parallel File Systems

Data Center Experience

HPC Background

Error Bar Analysis

Batching

Job description

Jobtailor is seeking a senior capacity optimization lead to manage GPU and large-scale compute workloads across multi-node training environments. You will own the capacity model, including node counts, assumptions, and error bars, and defend capacity decisions with real financial impact.

You will optimize workload mix, preemptible fractions, and checkpointing economics, while improving training throughput and data locality across InfiniBand/RDMA networks and parallel file systems.