Nvidia GPU Infrastructure Engineer / HPC Engineer

Larsen & Toubro Limited

Chennai

Hybrid

INR 900,000 - 1,300,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Larsen & Toubro (L&T) is seeking an experienced GPU infrastructure engineer to build and operate large-scale GPU compute pods for training and inference across a 10K GPU cluster. You will stand up clusters, manage partitioning, and integrate schedulers with device plugins.

The role emphasizes reliability, performance tuning (NCCL/UCX), incident response, and secure, scalable operations in a hybrid onsite environment.

Qualifications

  • 712 years building/operating HPC/AI GPU clusters at scale; deep CUDA/MIG/vGPU expertise.
  • Proven with H100/B-series GPUs, Slurm and/or Kubernetes device scheduling; NCCL/UCX performance tuning.
  • Experience with GPU device plugins, quotas, and multitenant GPU environments.

Responsibilities

  • Stand up multipod GPU clusters: rack/power/cooling; TOR/leaf connectivity; IB/Ethernet host configs.
  • Implement GPU partitioning (MIG/vGPU) and quota policies for multitenant environments.
  • Integrate cluster schedulers (Slurm/Kubernetes) with GPU device plugins, node feature discovery, and accounting/quotas.

Skills

CUDA/MIG/vGPU
HPC GPU clusters
Performance tuning
Incident response

Education

BE/B.Tech or equivalent in CS or ECE

Tools

Slurm
Kubernetes
NVIDIA DCGM/NVML
nvidia-smi
GPUDirect Storage
Helm/ArgoCD

Job description

Job Purpose

Build and operate largescale GPU compute pods to deliver predictable, highthroughput, lowlatency training and inference services across 10K GPU cluster.

Role Description
Key Responsibilities
  • Implementation
    • Stand up multipod GPU clusters (rack/power/cooling layouts; TOR/leaf connectivity; IB/Ethernet host configs).
    • Implement GPU partitioning (MIG/vGPU profiles) and quota policies for multitenant environments.
    • Integrate cluster schedulers (Slurm/Kubernetes) with GPU device plugins, node feature discovery, and accounting/quotas.
  • Operations
    • Own day2 operations across firmware/driver/DCGM/NVML lifecycles; execute change windows with zero/low downtime.
    • Capacity planning (GPU/CPU/Memory/NIC) and binpacking strategies for heterogeneous GPU SKUs.
  • Performance & Optimization
    • Tune NCCL/UCX, GPU clocks/persistence, GPU Direct Storage, NUMA/locality, and CUDA runtime parameters.
    • Drive benchmarking and acceptance (HPL, HPLAI, MLPerflike internal suites); track perf regressions with SLOs.
  • Reliability & Incident
    • Lead P0/P1 incident response for GPU, CUDA, or scheduler issues; perform rootcause and preventative actions.
    • Act as the primary technical lead during production outages, coordinating cross-functional teams across Networking, GPU Operations, Platform Engineering, Storage, and Application teams to restore services within SLA targets.
    • Perform detailed Root Cause Analysis (RCA) for network, GPU, CUDA, NCCL, RDMA, and scheduler-related failures, identifying underlying causes and implementing preventive and corrective actions.
    • Collaborate with platform and GPU engineering teams to resolve issues impacting CUDA jobs, Kubernetes scheduling, Slurm workload management, GPU resource allocation, and large-scale AI training environments.
    • Define golden images; implement node remediation (cordon/drain/reimage) and autohealing workflows.
  • Security & Compliance
    • Enforce GPU tenancy isolation (MIG, cgroup/device cgroup, mpsd), secure drivers/containers, SBOM and image scanning.
  • Documentation & Enablement
    • Publish runbooks, performance baselines, and application tuning guides for LLM training and inference.
Experience & Educational Requirements
Qualifications and Experience
EDUCATIONAL QUALIFICATIONS:

BE/B-Tech or equivalent with Computer Science or Electronics & Communication

RELEVANT EXPERIENCE:
  • 712 years building/operating HPC/AI GPU clusters at scale; deep CUDA/MIG/vGPU expertise.
  • Proven with H100/Bseries class GPUs, Slurm and/or Kubernetes device scheduling; NCCL/UCX performance tuning.
Tools / Tech

1. CUDA, NCCL, cuDNN, TensorRT; 2. DCGM/NVML, nvidia-smi; 3. Slurm, K8s (device plugin, DaemonSets, NFD); 4. Helm/ArgoCD; 6. GPUDirect Storage; 7. Prometheus/Grafana; 8. ELK/Splunk.

Certifications

NVIDIA Certified Professional (AI Infrastructure); Linux (RHCE/LPIC) preferred.

KPIs

GPU utilization %, queue waittime SLOs, failedjob rate, perf baseline adherence, MTTD/MTTR for GPU incidents.

Work Mode

Onsite/Hybrid; participates in change windows and critical afterhours events.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

GPU Infrastructure Engineer / HPC Engineer
GPU Infrastructure Engineer / HPC Engineer

L&T Technology Services Ltd. • Mumbai

On-site
INR 4,000,000 - 8,500,000
GPU Cluster Reliability Engineer
GPU Cluster Reliability Engineer

Ishan Technologies • Ahmedabad District

On-site
INR 1,200,000 - 1,800,000
HPC Engineer (Compute)
HPC Engineer (Compute)

World Wide Technology, Inc. • Mumbai

On-site
INR 1,500,000 - 4,000,000
Nvidia Network Engineer (InfiniBand / Ethernet)
Nvidia Network Engineer (InfiniBand / Ethernet)

Larsen & Toubro Limited • Chennai

On-site
INR 4,200,000 - 6,800,000
HPC Lead
HPC Lead

World Wide Technology, Inc. • India

On-site
INR 3,200,000 - 6,000,000
HPC Lead
HPC Lead

World Wide Technology, Inc. • Mumbai

On-site
INR 3,500,000 - 6,000,000
Senior HPC Cluster Engineer - AI, ML
Senior HPC Cluster Engineer - AI, ML

NVIDIA Corporation • Maharashtra

On-site
INR 3,000,000 - 5,400,000
Senior HPC Cluster Engineer - AI, ML
Senior HPC Cluster Engineer - AI, ML

NVIDIA Corporation • India

On-site
INR 3,000,000 - 6,000,000
Specialist - Cloud Engineering
Specialist - Cloud Engineering

LTM • Chennai

On-site
INR 3,500,000 - 6,000,000
Technology Platform Engineer
Technology Platform Engineer

Accenture PLC • Bengaluru

On-site
INR 2,500,000 - 4,500,000