Training / AI Infrastructure

Genesis AI; San Carlos (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Genesis AI seeks a senior ML infrastructure engineer to reduce wall-clock time to convergence by profiling and removing bottlenecks from data pipelines to GPU kernels. You will design and optimize distributed PyTorch training for multi-node GPU clusters, with emphasis on scalability and robustness.

Role requires deep experience in ML infrastructure, production-grade Python, and low-level CUDA/cuDNN/Triton mastery to optimize hardware–software interactions for maximum utilization.

Qualifications

• 8+ years in distributed systems, ML infra, or HPC.
• Production-grade Python expertise.
• Expert CUDA/cuDNN/Triton and CPU–GPU tuning.
• Experience with PyTorch training jobs and parallelism.
• Proven ability to optimize hardware–software for utilization.

Responsibilities

• Profile and remove bottlenecks across training stack.
• Design and optimize distributed PyTorch systems for multi-node GPUs.
• Implement low-level CUDA kernels and integrate with frameworks.
• Optimize CPU/GPU balance, memory, throughput, and networking.
• Develop monitoring and debugging tools for large-scale runs.

Skills

Distributed systems

Python

CUDA optimization

PyTorch training

Kernel optimization

Tools

CUDA

cuDNN

Triton

PyTorch

Job description

What You’ll Do

• Drive down wall-clock time to convergence by profiling and eliminating bottlenecks across the foundation model training stack stack, from data pipelines to GPU kernels
• Design, build, and optimize distributed training systems (PyTorch) for multi-node GPU clusters, ensuring scalability, robustness, and high utilization
• Implement efficient low-level code (CUDA, cuDNN, Triton, custom kernels) and integrate it seamlessly into high-level training frameworks
• Optimize workloads for hardware efficiency: CPU/GPU compute balance, memory management, data throughput, and networking
• Develop monitoring and debugging tools for large-scale runs, enabling rapid diagnosis of performance regressions and failures

What You’ll Bring

• Deep experience in distributed systems, ML infrastructure, or high-performance computing (8+ years)
• Production-grade expertise in Python
• Low-level performance mastery: CUDA/cuDNN/Triton, CPU–GPU interactions, data movement, and kernel optimization
• Scaling at the frontier: experience with PyTorch and training jobs using data, context, pipeline, and model parallelism
• System-level mindset with a track record of tuning hardware–software interactions for maximum utilization