Senior AI Infra Engineer - GPU & HPC Reliability

Nscale; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

Nscale is seeking a Senior AI Infrastructure Support Engineer for APAC to join the Support and Operations team. You will serve as a senior escalation point, diagnosing GPU node faults, ensuring fabric health, and driving incident resolution across InfiniBand and RoCE fabrics.

The role demands 6+ years in infra/ops with 2–3+ years in GPU/HPC contexts, strong Linux skills, and hands-on with NVIDIA tools, RDMA, and automation. Travel to client sites may be required.

Qualifications

• 6+ years in infrastructure, operations, or support engineering in production environments.
• 2–3+ years hands-on with GPU, HPC, or large-scale data centre estates, ideally customer-facing.
• Able to explain complex technical detail clearly to customers and stakeholders.
• GPU platforms experience; NVIDIA preferred with driver/firmware/runtime knowledge.
• Experience with InfiniBand/RoCE fabrics and related diagnostics.
• Slurm operations for large multi-GPU jobs; container tools optional.
• Linux systems engineering across compute, storage, and network layers.
• Familiar with BMC/Redfish and bare-metal provisioning workflows.
• Strong networking fundamentals (L2/L3, routing, VLANs, VXLAN).
• Observability and incident response; dashboards and runbooks.
• Ability to author changes with risk assessment and backout plans.
• SRE-style automation and scripting for diagnostics and tooling.
• Data centre fundamentals and onsite travel readiness.

Responsibilities

• Join the Support duty rotation as a senior escalation point for incidents and changes.
• Diagnose GPU node faults across driver, firmware, and hardware layers; perform RMA-oriented analysis.
• Own east-west fabric health; run diagnostics and validate topology on InfiniBand/RoCE fabrics.
• Investigate data-path issues on high-performance storage and storage-network interactions.
• Lead structured investigations; perform root cause analysis and long-term fixes.
• Design and execute changes in live customer environments with risk assessment.
• Improve dashboards, alerts, and runbooks to prevent repeats of incidents.
• Record tickets with clear, customer-impact statements and detailed notes.
• Develop automation scripts to reduce toil and manual intervention.
• Act as a key escalation point; mentor engineers and share knowledge.
• Contribute to training content, workshops, and PR reviews.
• Travel to Nscale or customer sites for onsite technical support as needed.

Skills

GPU platforms

nvidia-smi/DCGM

XID/error interpretation

RDMA fabrics

InfiniBand/RoCE

SLURM scheduling

BMC/Redfish

Linux systems engineering

Automation scripting

Incident response

Runbooks/monitoring

Open-source collaboration

Tools

Prometheus

Grafana

MAAS

NetBox

Redfish

Python

Bash

Git

NCCL

Job description

Nscale is seeking a Senior AI Infrastructure Support Engineer for APAC to join the Support and Operations team. You will serve as a senior escalation point, diagnosing GPU node faults, ensuring fabric health, and driving incident resolution across InfiniBand and RoCE fabrics.

The role demands 6+ years in infra/ops with 2–3+ years in GPU/HPC contexts, strong Linux skills, and hands-on with NVIDIA tools, RDMA, and automation. Travel to client sites may be required.