Senior HPC Cluster Engineer - AI, ML

NVIDIA Corporation; Bengaluru; Hybrid; INR 4,000,000 - 7,000,000; Full time

Benefits offered by this job

NVIDIA employee benefits

Job summary

NVIDIA Bengaluru is hiring a Senior AI/ML HPC Cluster Engineer to lead the management of large-scale AI/HPC infrastructure in the MARS team. You will oversee production clusters, drive reliability, and collaborate with global teams to optimize performance and user satisfaction.

You will need 5+ years designing and operating big compute systems, with expertise in Slurm and related schedulers, Linux admin, and scripting in Python/Bash. GPU-accelerated workloads experience is a plus.

Qualifications

• Bachelor’s degree in Computer Science, Electrical Engineering or related field.
• Minimum 5 years of experience designing and operating large-scale compute infrastructure.
• Experience with AI/HPC advanced job schedulers, such as Slurm, K8s, PBS, RTDA, BCM, or LSF.
• Proficient in administering Centos/RHEL and/or Ubuntu Linux distributions.
• Solid understanding of cluster configuration tools (BCM, Terraform, Ansible, Puppet, Salt, etc.), container technologies (Docker, Singularity, Podman, Shifter, Charliecloud), Python programming, and bash scripting.
• Applied experience with AI/HPC workflows that use MPI.
• Experience analyzing and tuning performance for a variety of AI/HPC workloads.
• Passion for continual learning and staying ahead of emerging technologies in HPC/AI infra.

Responsibilities

• Provide leadership in systems administration and service delivery on our AI/HPC fleet by coordinating system upgrades, responding to incidents, and delivering reliability improvements.
• Collaborate closely with global teams to deliver a world class user experience in AI and HPC research.
• Own day-to-day operations of production AI/HPC clusters, ensuring system health, user satisfaction, and efficient resource utilization.
• Develop and improve our ecosystem around GPU-accelerated computing including scalable automation solutions.
• Build and maintain heterogeneous AI/ML clusters on-premises and in the cloud.
• Create and cultivate customer and cross-team relationships to meet user evolving needs.
• Support our researchers to run workloads including performance analysis and optimizations.
• Analyze and optimize cluster efficiency, job fragmentation, and GPU waste to meet internal SLA targets.
• Conduct root cause analysis and implement corrective actions; proactively fix issues before they occur.
• Lead SEV triage and postmortems for reliability incidents affecting users or infrastructure.
• Participate in on-call rotation and incident response for critical production GPU clusters.

Skills

Slurm

K8s

PBS

RTDA

BCM

LSF

Python scripting

Bash scripting

MPI

CUDA / GPU workflows

Education

Bachelor’s degree in Computer Science, Electrical Engineering or related field

Tools

Terraform

Ansible

Puppet

Salt

Docker

Singularity

Podman

Shifter

Charliecloud

Job description

## Senior HPC Cluster Engineer - AI, MLApplylocations: India, Bengaluru: India, Remote: India, Punetime type: Full timeposted on: Posted Todayjob requisition id: JR2021817NVIDIA is a pioneer in accelerated computing, known for inventing the GPU and driving breakthroughs in gaming, computer graphics, high-performance computing, and artificial intelligence. Our technology powers everything from generative AI to autonomous systems, and we continue to shape the future of computing through innovation and collaboration. Within this mission, our team, Managed AI Superclusters (MARS) builds and scales the infrastructure, platforms, and tools that enable researchers and engineers to develop the next generation of AI/ML systems. By joining us, you’ll help design solutions that power some of the world’s most advanced computing workloads.NVIDIA is looking for a Senior AI/ML HPC Cluster Engineer to join our MARS team. You will provide leadership and strategic guidance on the management of large-scale HPC systems including the deployment of compute, networking, and storage. You will be working with a team of passionate and skilled engineers across NVIDIA that are continuously working to provide better tools to build and manage this infrastructure. Ideal candidate is strong in building and maintaining distributed clusters, driving improvements, and has the ability to understand researcher computing needs.**What you'll be doing:*** Provide leadership in systems administration and service delivery on our AI/HPC fleet by coordinating system upgrades, responding to incidents, and delivering reliability improvements.* Collaborate closely with global teams to deliver a world class user experience in AI and HPC research.* Own day-to-day operations of production AI/HPC clusters, ensuring system health, user satisfaction, and efficient resource utilization.* Develop and improve our ecosystem around GPU-accelerated computing including developing scalable automation solutions.* Build and maintain heterogeneous AI/ML clusters on-premises and in the cloud.* Create and cultivate customer and cross-team relationships to meet user evolving user needs.* Support our researchers to run their workloads including performance analysis and optimizations* Analyze and optimize cluster efficiency, job fragmentation, and GPU waste to meet internal SLA targets.* Conduct root cause analysis and suggest corrective action Proactively find and fix issues before they occur.* Lead SEV triage and postmortems for reliability incidents affecting users or infrastructure.* Participate in on-call rotation and incident response for critical production GPU clusters.**What we need to see:*** Bachelor’s degree in Computer Science, Electrical Engineering or related field or equivalent experience* Minimum 5 years of experience designing and operating large scale compute infrastructure* Experience with AI/HPC advanced job schedulers, such as Slurm, K8s, PBS, RTDA, BCM, or LSF* Proficient in administering Centos/RHEL and/or Ubuntu Linux distributions* Solid understanding of cluster configuration management tools (BCM, Terraform, Ansible, Puppet, Salt, etc.), container technologies (Docker, Singularity, Podman, Shifter, Charliecloud), Python programming, and bash scripting.* Applied experience with AI/HPC workflows that use MPI* Experience analyzing and tuning performance for a variety of AI/HPC workloads.* Passion for continual learning and staying ahead of emerging technologies and effective approaches in the HPC and AI/ML infrastructure fields.**Ways to stand out from the crowd:*** Background with NVIDIA GPUs, CUDA Programming, NCCL and MLPerf benchmarking* Experience with AI/ML concepts, algorithms, models, and frameworks (PyTorch, Tensorflow)* Experience with InfiniBand with IPoIB and RDMA* Understanding of fast, distributed storage systems such as Lustre and GPFS for AI/HPC workloadsNVIDIA pioneered accelerated computing. Today, our AI infrastructure powers global intelligence, transforming every industry.