AI GPU Cluster Networking Engineer (InfiniBand/RoCE)

Aquila Hash, Inc.; Buffalo (NY); On-site; USD 110,000 - 160,000; Full time

Job summary

Aquila Hash, Inc. in Buffalo, NY, is seeking an AI GPU Cluster Operations – Network Engineer to lead operations, maintenance and troubleshooting of high‑performance networking infrastructure for large‑scale AI GPU clusters.

The role centers on InfiniBand, RoCE and data center IP networks, with emphasis on GPU servers, Linux and overall AI cluster architecture. You will monitor health, resolve incidents and work with vendors to drive issues to resolution.

Qualifications

• Degree or relevant educational background in Computer Science, Information Technology, Networking, Engineering or a related field.
• 3+ years of hands-on network operations, data center networking or infrastructure experience.
• Hands-on experience operating InfiniBand and/or RoCE networks in GPU, AI, HPC or large-scale data center environments.
• Experience with NVIDIA/Mellanox InfiniBand or high-speed Ethernet networking equipment is strongly preferred.
• Understanding of InfiniBand architecture, Subnet Management, partitions, routing, link states and performance counters.
• Strong understanding of TCP/IP and data center networking.

Responsibilities

• Operate and maintain high-performance GPU cluster networks, including InfiniBand and RoCE, and standard data center IP networks.
• Manage and troubleshoot NVIDIA/Mellanox InfiniBand switches, high-speed Ethernet switches, NICs/HCAs, optical transceivers and related infrastructure.
• Monitor network health and identify link degradation, congestion, packet loss, bandwidth or latency issues.
• Perform InfiniBand health checks and diagnostics using ibstat, ibdiagnet and NVIDIA/Mellanox utilities.
• Support InfiniBand Subnet Management, partition configuration, routing, and performance-counter analysis.
• Troubleshoot RoCE environments, including connectivity and lossless Ethernet issues.
• Support data center IP network operations (BGP, OSPF, VLAN, MLAG, TCP/IP).
• Troubleshoot GPU node connectivity and determine Root Cause across server, NIC/HCA, optics, cabling or fabric.
• Collaborate with GPU Cluster Operations Engineers to resolve NCCL failures and performance degradation.
• Monitor via Prometheus, SNMP, Grafana and related platforms; develop monitoring and diagnostic capabilities.
• Perform approved network configuration changes and SOP-driven maintenance/upgrades.
• Support network capacity planning and expansion of GPU cluster infrastructure.
• Automate tasks using Python, Shell, Ansible and/or Terraform.
• Participate in incident response, post-incident reviews and RCA.
• Maintain network diagrams, configuration docs and troubleshooting guides.
• Coordinate with NVIDIA/Mellanox, server OEMs and vendors to resolve hardware/network issues.
• Support hardware replacement and RMA activities for switches, NICs/HCAs and optics.

Skills

InfiniBand

RoCE

Linux

TCP/IP

Prometheus

Grafana

Python

Shell

Ansible

Terraform

BGP

OSPF

VLAN

MLAG

NCCL

Education

Degree in Computer Science / IT / Networking / Engineering or related field

Tools

Mellanox InfiniBand

ibstat

ibdiagnet

NICs/HCAs

100G/200G/400G NICs

Job description

Aquila Hash, Inc. in Buffalo, NY, is seeking an AI GPU Cluster Operations – Network Engineer to lead operations, maintenance and troubleshooting of high‑performance networking infrastructure for large‑scale AI GPU clusters.

The role centers on InfiniBand, RoCE and data center IP networks, with emphasis on GPU servers, Linux and overall AI cluster architecture. You will monitor health, resolve incidents and work with vendors to drive issues to resolution.