Senior AI Infrastructure Engineer - Distributed GPU Training

Genesis; Greater London; On-site; GBP 110,000 - 170,000; Full time

Job summary

Genesis in London is seeking an senior ML infrastructure engineer to drive down wall-clock time in foundation model training. You will design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters, ensuring scalability and high utilization.

You’ll implement low-level code (CUDA, cuDNN, Triton) and integrate into training frameworks, optimize CPU/GPU balance, memory and data throughput, and develop monitoring tools for large-scale runs.

Qualifications

• 8+ years in distributed systems, ML infrastructure or high-performance computing.
• Production-grade Python proficiency essential.
• Low-level performance mastery across CUDA/cuDNN/Triton and CPU–GPU interactions.
• Experience with PyTorch training jobs using data, context, pipeline, and model parallelism.

Responsibilities

• Drive down wall-clock time to convergence by profiling and removing bottlenecks across the foundation model training stack.
• Design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters with scalability and high utilization.
• Implement efficient low-level code (CUDA, cuDNN, Triton, custom kernels) and integrate into high-level training frameworks.
• Optimize workloads for hardware efficiency: CPU/GPU balance, memory management, data throughput, networking.
• Develop monitoring and debugging tools for large-scale runs to diagnose performance regressions and failures.

Skills

Distributed systems experience

Python programming

Performance optimization

Large-scale ML experience

Tools

CUDA

cuDNN

Triton

PyTorch

Job description

Genesis in London is seeking an senior ML infrastructure engineer to drive down wall-clock time in foundation model training. You will design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters, ensuring scalability and high utilization.

You’ll implement low-level code (CUDA, cuDNN, Triton) and integrate into training frameworks, optimize CPU/GPU balance, memory and data throughput, and develop monitoring tools for large-scale runs.