Lead AI Training Infrastructure Engineer

Genesis; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

Genesis seeks a senior ML infrastructure engineer to design and optimize distributed training systems on multi-node GPU clusters. You will push wall-clock time to convergence by profiling, refining data pipelines, and low-level kernels in CUDA/CuDNN/Triton.

You will build scalable PyTorch-based workflows, balance CPU/GPU workloads, and develop monitoring tools to diagnose performance regressions at scale.

Qualifications

• Deep experience in distributed systems, ML infrastructure, or high-performance computing (8+ years).
• Production-grade expertise in Python.
• Low-level performance mastery: CUDA/cuDNN/Triton, CPU-GPU interactions, data movement, and kernel optimization.
• Scaling at the frontier: experience with PyTorch and training jobs using data, context, pipeline, and model parallelism.
• System-level mindset with a track record of tuning hardware-software interactions for maximum utilization.

Responsibilities

• Drive down wall-clock time to convergence by profiling and eliminating bottlenecks across the foundation model training stack, from data pipelines to GPU kernels.
• Design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters, ensuring scalability, robustness, and high utilization
• Implement efficient low-level code (CUDA, cuDNN, Triton, custom kernels) and integrate it seamlessly into high-level training frameworks
• Optimize workloads for hardware efficiency: CPU/GPU compute balance, memory management, data throughput, and networking
• Develop monitoring and debugging tools for large-scale runs, enabling rapid diagnosis of performance regressions and failures

Skills

Distributed systems

Python

CUDA

cuDNN

Triton

CPU-GPU interactions

Kernel optimization

PyTorch

Performance tuning

Tools

PyTorch

Job description

Genesis seeks a senior ML infrastructure engineer to design and optimize distributed training systems on multi-node GPU clusters. You will push wall-clock time to convergence by profiling, refining data pipelines, and low-level kernels in CUDA/CuDNN/Triton.

You will build scalable PyTorch-based workflows, balance CPU/GPU workloads, and develop monitoring tools to diagnose performance regressions at scale.