Network Performance Engineer – GPU/HPC Infrastructure

Ishan Technologies; Ahmedabad District; On-site; INR 2,400,000 - 5,400,000; Full time

Job summary

Ishan Technologies is seeking an experienced Network Performance Engineer to build, operate, optimise and maintain a high-performance GPU network supporting a 1000+ GPU cluster. You will work on RoCEv2/Ethernet and InfiniBand fabrics, L2/L3 escalation, and performance benchmarking.

The role emphasizes RDMA, RCCL/NCCL/MPI benchmarking, network automation with Python/Ansible, and collaboration with NOC and MLOps teams to improve AI workload performance.

Qualifications

• Hands-on data centre networking experience required.
• Experience with RoCEv2 or InfiniBand essential.
• Strong knowledge of BGP, EVPN-VXLAN and leaf-spine.
• Familiarity with RDMA, GPUDirect RDMA and SR-IOV.
• Experience with NVIDIA ConnectX/Spectrum-X or AMD Pensando Pollara.
• Awareness of VLAN, PKeys and tenant isolation.
• Willingness to participate in 24x7 on-call.
• Excellent troubleshooting and analytical skills.

Responsibilities

• Install, configure and operate high-speed backend GPU networks.
• Manage front-end, storage and out-of-band networks.
• Configure switches, NICs, transceivers and fibre links.
• Maintain network topology, cables and documentation.
• Perform RCCL/NCCL/MPI benchmarking and perftest tests.
• Collaborate with NOC and compute teams to optimise workloads.

Skills

Data Centre Networking

RoCEv2/InfiniBand

BGP-EVPN-VXLAN

PFC-ECN-QoS

RDMA-GPUDirect SR-IOV

NVIDIA ConnectX Spectrum-X

RCCL/NCCL/MPI benchmarking

NVIDIA UFM/InfiniBand mgmt

VLAN/VRF/PKeys isolation

400G/800G links

Python/Ansible

Prometheus/Grafana/SNMP telemetry

Education

Bachelor's degree in Engineering or Technology

CS/IT/Electronics/Telecommunications specialization

Data centre networking certifications

Job description

Ishan Technologies is looking for an experienced Network Performance Engineer to build, operate, optimise and maintain the high-performance network infrastructure supporting Saksham AI Services' 1000+ GPU cluster. The role will focus on backend GPU networking, high-speed Ethernet/RoCE and InfiniBand fabrics, front-end connectivity, storage and management networks. The ideal candidate should have strong hands-on experience in data centre networking, network performance optimisation, RDMA technologies, congestion control and collective communication benchmarking. The selected candidate will also act as an L2/L3 escalation point for network and performance-related issues raised by Ishan Technologies' 24x7 NOC team.

Key Responsibilities

• Install, configure and operate high-speed backend GPU networks using RoCEv2/Ethernet or InfiniBand.
• Manage front-end, storage and out-of-band management networks.
• Configure and maintain switches, network interface cards (NICs), transceivers and fibre connectivity.
• Work on rail-optimised leaf-spine network architecture.
• Maintain network topology, cable mapping and infrastructure documentation.

2. Network Performance & Optimisation

• Configure and optimise BGP, EVPN-VXLAN and leaf-spine network architecture.
• Implement congestion control mechanisms, including PFC, ECN/DCQCN, adaptive routing and QoS.
• Monitor network bandwidth, latency, packet loss, link flaps and congestion.
• Troubleshoot GPU-to-GPU communication and collective performance issues.
• Ensure full-bandwidth network performance across different workload and cluster sizes.

3. RoCE, InfiniBand & RDMA Networking

• Manage RoCEv2 and InfiniBand fabrics for GPU/HPC environments.
• Work with NVIDIA UFM and InfiniBand management tools.
• Configure and troubleshoot RDMA drivers, SR-IOV and GPUDirect RDMA.
• Work with NVIDIA ConnectX and AMD Pensando Pollara networking technologies.
• Maintain VLAN/VRF segmentation, PKeys and fabric-level tenant isolation.
• Perform collective communication benchmarking using RCCL, NCCL, MPI and related testing tools.
• Conduct RDMA performance testing using perftest, ib_write_bw and ib_read_bw.
• Work with nccl-tests and rccl-tests to validate network performance.
• Understand how data, tensor and expert parallelism impact network utilisation.
• Investigate slow-running AI jobs, communication bottlenecks and straggler GPUs.
• Collaborate with compute and MLOps teams to optimise AI workload performance.

5. Network Monitoring & Automation

• Develop network monitoring dashboards using Prometheus and Grafana.
• Monitor switch health, link errors, optics performance and network congestion.
• Automate network configurations, health checks and compliance activities using Python and Ansible.
• Maintain network performance baselines and conduct acceptance testing after infrastructure changes.
• Support Kubernetes RDMA networking using Multus, SR-IOV device plugins and network operators.

6. Incident Management & Operations

• Handle L2/L3 network and performance escalations from the 24x7 NOC team.
• Lead network incident investigations and root cause analysis.
• Coordinate with customers, solution architects, vendors and internal engineering teams.
• Plan and execute switch, NIC and optics firmware upgrades under change management.
• Maintain network runbooks, technical documentation and operational standards.
• Support internet edge connectivity, peering and private customer network links.

Required Technical Skills

• Strong hands-on experience in Data Centre Networking.
• Experience with RoCEv2 or InfiniBand networking.
• Strong understanding of BGP, EVPN-VXLAN and leaf-spine architecture.
• Knowledge of PFC, ECN/DCQCN, QoS and adaptive routing.
• Experience with RDMA, GPUDirect RDMA and SR-IOV.
• Hands-on experience with NVIDIA ConnectX, Spectrum-X or AMD Pensando Pollara.
• Knowledge of RCCL, NCCL, MPI and collective communication benchmarking.
• Experience with NVIDIA UFM and InfiniBand management.
• Understanding of VLAN, VRF, PKeys and network tenant isolation.
• Experience with 400G/800G networking, optics and fibre connectivity.
• Working knowledge of Python, Ansible and network automation.
• Experience with Prometheus, Grafana, SNMP and streaming telemetry.

Educational Qualifications

• Bachelor's degree in Engineering or Technology.
• Preferred specialisations: Computer Science, Information Technology, Electronics, Telecommunications or related disciplines.
• Relevant data centre networking certifications will be an added advantage.

Preferred Certifications:

• Arista ACE.
• NVIDIA Networking (InfiniBand/Spectrum-X) certifications.

Experience Requirements

• 6–10 years of overall professional experience.
• Minimum 4+ years of relevant experience in Data Centre Networking for GPU, HPC or hyperscale environments.
• Hands-on experience with RoCEv2 or InfiniBand fabrics is essential.
• Experience with large-scale leaf-spine networks, cabling and optics is essential.
• Experience in RCCL/NCCL or MPI performance optimisation is preferred.
• Exposure to AMD Pensando, NVIDIA Spectrum-X or ConnectX networking is preferred.
• Strong troubleshooting and analytical skills.
• Hands-on approach to network engineering and performance optimisation.
• Ability to handle critical production incidents under pressure.
• Strong ownership and root cause analysis capabilities.
• Effective communication with customers, NOC teams and technical stakeholders.
• Willingness to participate in a 24x7 on-call support rotation.
• Ability to learn and adapt to evolving AI and GPU networking technologies.

Key Success Outcomes

• Stable, high-performance GPU networking infrastructure.
• Full-bandwidth collective communication across cluster workloads.
• Verified network isolation between multiple customers and tenants.
• Improved network performance and reduced communication bottlenecks.
• Faster resolution of network incidents and performance escalations.
• Standardised, automated and well-documented network operations.