GPU Cluster Reliability Engineer

Ishan Technologies; Ahmedabad District; In loco; INR 1.200.000 - 1.800.000; Tempo pieno

Descrizione del lavoro

Ishan Technologies is seeking an experienced Cluster Reliability Engineer to oversee a 1000+ GPU compute cluster for Saksham AI Services. You will manage GPU servers, Linux, Slurm, and Kubernetes, ensuring high availability, performance, and scalable AI workloads.

The role serves as an L2/L3 escalation point, coordinating with a 24x7 NOC, platform engineers and partners. Strong automation and Python/Bash scripting are essential for reliability improvements.

Competenze

• Bachelor's degree in Engineering or Technology; CS/IT/Electronics preferred.
• Linux, Kubernetes and GPU computing certifications are a plus.
• Experience with Slurm and multi-GPU HPC environments.
• Hands-on Linux sysadmin experience on GPU clusters.
• Familiarity with data-center hardware, firmware and drivers.

Mansioni

• Rack, configure and maintain GPU servers, BIOS and firmware.
• Manage Linux OS images, GPU drivers and software libraries.
• Administer Slurm and Kubernetes for AI workloads; optimize resources.
• Develop automation scripts for provisioning and monitoring.
• Lead incident investigations and post-incident reviews.
• Maintain monitoring dashboards with Prometheus/Grafana and alerting.

Conoscenze

Linux administration

GPU servers management

Slurm administration

Kubernetes administration

Python scripting

Bash scripting

Ansible or Terraform

Troubleshooting

On-call incident handling

Formazione

Bachelor's degree in Engineering or Technology

CS/IT/Electronics specialization

Strumenti

Docker

Apptainer

GPU Operator

Helm

Kubeflow

KubeRay

Prometheus

Grafana

Loki/ELK

Descrizione del lavoro

Ishan Technologies is looking for an experienced Cluster Reliability Engineer to manage and maintain the compute infrastructure of Saksham AI Services' 1000+ GPU cluster.

The role involves managing GPU servers, Linux infrastructure, Slurm and Kubernetes environments, cluster reliability, automation and MLOps operations. The selected candidate will play a key role in ensuring high availability, performance, scalability and reliability of AI training and inference workloads.

The position will also serve as an L2/L3 technical escalation point, working closely with the 24x7 NOC team, platform engineers, customers and technology partners.

Key Responsibilities

1. GPU Infrastructure & Server Management

• Receive, rack, configure and maintain GPU servers.
• Manage BIOS, firmware, Linux OS images, GPU drivers and software libraries.
• Perform GPU server burn-in, hardware diagnostics and acceptance testing.
• Maintain consistent server configurations and manage firmware and security updates.
• Coordinate with OEM vendors for hardware faults, replacements and RMAs.
• Administer and maintain Slurm and Kubernetes environments for AI training and inference workloads.
• Manage Slurm controllers, partitions, QoS, fairshare and resource allocation.
• Configure Kubernetes GPU Operator, device plugins, Helm and related operators.
• Manage cluster upgrades, patching, tenant onboarding and scheduling.
• Ensure stable and efficient multi-tenant GPU cluster operations.

3. Cluster Reliability & Monitoring

• Develop and maintain GPU and node health checks.
• Monitor GPU errors, ECC events, RAS events and hardware failures.
• Implement automated fault detection and remediation mechanisms.
• Develop monitoring dashboards and alerts using Prometheus, Grafana, Loki or ELK.
• Maintain cluster availability in line with the 99.9% uptime commitment.
• Maintain operational logs and monitoring records in accordance with applicable compliance requirements.
• Build and maintain optimized framework images for PyTorch, JAX and vLLM.
• Develop ready-to-use job templates for AI training and inference.
• Conduct workload benchmarks and GPU performance testing.
• Support distributed AI workloads, including data, tensor and pipeline parallelism.
• Provide technical assistance to customers and solution architects on workload performance and tooling.
• Handle L2/L3 technical escalations from the 24x7 NOC team.
• Lead incident investigations, root cause analysis and post-incident reviews.
• Develop automation scripts using Python and Bash.
• Maintain operational runbooks and auto-remediation procedures.
• Track capacity, GPU utilisation, goodput and service-level performance.
• Identify recurring issues and implement long-term corrective actions.

Required Technical Skills

• Strong hands-on experience in Linux System Administration.
• Experience managing GPU servers, GPU drivers, firmware, BIOS and kernel configurations.
• Hands-on experience with multi-GPU accelerator servers.
• Strong knowledge of Slurm administration and workload scheduling.
• Experience with Kubernetes administration for GPU workloads.
• Knowledge of GPU Operator, Helm, Kubeflow and KubeRay.
• Familiarity with GPU monitoring, ECC/RAS events and vendor diagnostic tools.
• Experience with Prometheus, Grafana, Loki or ELK.
• Working knowledge of PyTorch, JAX, vLLM and GPU collective communication libraries.
• Experience with Docker, Apptainer and container image management.
• Automation experience using Python, Bash and Ansible or Terraform.
• Familiarity with bare-metal provisioning tools such as MAAS, Ironic, Metal3 or Warewulf.

Educational Qualifications

• Bachelor's degree in Engineering or Technology.
• Preferred specialisations: Computer Science, Information Technology, Electronics or related disciplines.
• Relevant certifications in Linux, Kubernetes or GPU computing will be an added advantage.

Preferred Certifications:

• Red Hat RHCSA/RHCE or Linux Foundation Certified System Administrator (LFCS).
• GPU vendor training or certifications.

Experience Requirements

• 4–8 years of overall IT infrastructure or engineering experience.
• Minimum 2+ years of relevant experience in GPU/HPC cluster operations, Site Reliability Engineering (SRE) or MLOps.
• Hands-on experience with Linux GPU servers and Slurm and/or Kubernetes is essential.
• Experience supporting production environments with on-call and incident management responsibilities.
• Experience with multi-vendor data-centre GPU platforms will be an advantage.
• Exposure to distributed AI training and inference workloads is preferred.

Preferred Additional Exposure

• NVIDIA GPU ecosystem and related management tools.
• ClusterMAX standards and CMAX audit tool.
• Public cloud platforms such as GCP or AWS.
• HPC infrastructure and large-scale AI compute environments.
• Strong troubleshooting and problem-solving abilities.
• Hands-on technical ownership and attention to detail.
• Ability to handle critical production incidents under pressure.
• Strong communication and coordination skills.
• Willingness to participate in a 24x7 on-call support rotation.
• Ability to learn and adapt to evolving GPU, HPC and AI technologies.