Training / AI Infrastructure

Genesis AI; United States; Hybrid; USD 190,000 - 240,000; Full time

Job summary

Genesis AI is seeking an ultra-senior ML infrastructure engineer in the United States to accelerate foundation model training. You will profile bottlenecks from data pipelines to GPU kernels and craft scalable PyTorch-based systems for multi-node clusters.

You will write low-level CUDA/CuDNN/Triton kernels, optimize CPU-GPU data movement, and build robust tooling to monitor, diagnose, and fix performance regressions across large-scale runs.

Qualifications

• 8+ years of experience in distributed systems, ML infrastructure, or high-performance computing.
• Proficiency in Python with production-grade capabilities.
• Strong understanding of low-level optimization: CUDA/cuDNN/Triton, CPU-GPU interactions, data movement, kernel tuning.

Responsibilities

• Drive down wall-clock time to convergence by profiling and eliminating bottlenecks across the foundation model training stack—from data pipelines to GPU kernels.
• Design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters, ensuring scalability, robustness, and high utilization.
• Implement efficient low-level code (CUDA, cuDNN, Triton, custom kernels) and integrate it into high-level training frameworks.
• Optimize workloads for hardware efficiency: CPU-GPU compute balance, memory management, data throughput, and networking.
• Develop monitoring and debugging tools for large-scale runs, enabling rapid diagnosis of performance regressions and failures.

Skills

Distributed systems

HPC

Python

Tools

CUDA

cuDNN

Triton

PyTorch

Job description

What You’ll Do

• Drive down wall-clock time to convergence by profiling and eliminating bottlenecks across the foundation model training stack stack, from data pipelines to GPU kernels
• Design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters, ensuring scalability, robustness, and high utilization
• Implement efficient low-level code (CUDA, cuDNN, Triton, custom kernels) and integrate it seamlessly into high-level training frameworks
• Optimize workloads for hardware efficiency: CPU-GPU compute balance, memory management, data throughput, and networking
• Develop monitoring and debugging tools for large-scale runs, enabling rapid diagnosis of performance regressions and failures

What You’ll Bring

• Deep experience in distributed systems, ML infrastructure, or high-performance computing (8+ years)
• Production-grade expertise in Python
• Low-level performance mastery: CUDA/cuDNN/Triton, CPU-GPU interactions, data movement, and kernel optimization
• Scaling at the frontier: experience with PyTorch and training jobs using data, context, pipeline, and model parallelism
• System-level mindset with a track record of tuning hardware-software interactions for maximum utilization