Senior InfiniBand Network Architect: UFM & HPC Infra

lightningai

New York (NY)

On-site

USD 150,000 - 210,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Lightning AI is seeking an experienced Senior Network Engineer (InfiniBand / UFM) to design, deploy, and operate high‑performance AI Factory networking infrastructure for large GPU clusters. You will build NVIDIA InfiniBand fabrics powering training and inference environments using UFM, NCCL, and RoCEv2.

You will automate provisioning with Python and Ansible, monitor health with Prometheus and Grafana, and collaborate with AI platform, storage, and systems teams to scale our infrastructure while

Qualifications

  • 7+ years of data center networking experience.
  • 3+ years supporting NVIDIA InfiniBand environments.
  • Hands-on experience with NVIDIA UFM Enterprise.
  • Experience deploying and operating Quantum and Quantum-2 InfiniBand switches.
  • Strong Linux administration experience (Ubuntu).
  • Experience with Python and Ansible for automation.
  • Deep understanding of Layer 2 and Layer 3 networking.
  • Experience with BGP, EVPN, VXLAN.
  • Experience with packet captures and troubleshooting using tcpdump, Wireshark, ibdiagnet tools.

Responsibilities

  • Design, deploy, and maintain large-scale NVIDIA InfiniBand fabrics supporting AI/ML GPU clusters.
  • Deploy and administer NVIDIA Unified Fabric Manager (UFM) Enterprise for monitoring, provisioning, telemetry, and fabric health.
  • Configure and optimize NVIDIA Quantum and Quantum-2 InfiniBand switches.
  • Troubleshoot fabric performance issues impacting NCCL, MPI, GPUDirect RDMA, and AI training jobs.
  • Implement and validate fat-tree, Dragonfly+, Clos, and spine-leaf network architectures.
  • Perform firmware lifecycle management for InfiniBand switches, adapters (HCAs), and UFM infrastructure.
  • Optimize congestion control, adaptive routing, QoS, and traffic engineering for high-performance GPU communication.
  • Work closely with AI platform, GPU infrastructure, storage, and systems engineering teams to deploy scalable AI Factory environments.
  • Automate network provisioning using Python, Ansible, Git, REST APIs, and Infrastructure-as-Code methodologies.
  • Monitor network health using UFM telemetry, Prometheus, Grafana, and other observability platforms.
  • Support high availability, maintenance windows, incident response, root cause analysis, and capacity planning.
  • Participate in architecture reviews and define networking standards for AI infrastructure.

Skills

Data center networking
NVIDIA InfiniBand
NVIDIA UFM
InfiniBand switches
Linux administration (Ubuntu)
Python automation
Ansible automation
L2/L3 networking
BGP/EVPN/VXLAN
Packet capture (tcpdump/Wireshark/ibdi

Tools

NVIDIA UFM Enterprise
Prometheus
Grafana
tcpdump/Wireshark
ibdiagnet
Python
Ansible

Job description

Lightning AI is seeking an experienced Senior Network Engineer (InfiniBand / UFM) to design, deploy, and operate high‑performance AI Factory networking infrastructure for large GPU clusters. You will build NVIDIA InfiniBand fabrics powering training and inference environments using UFM, NCCL, and RoCEv2.

You will automate provisioning with Python and Ansible, monitor health with Prometheus and Grafana, and collaborate with AI platform, storage, and systems teams to scale our infrastructure while

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior InfiniBand Network Engineer (AI Clusters)
Senior InfiniBand Network Engineer (AI Clusters)

Lightning AI • San Francisco (CA), Seattle (WA), New York (NY)

On-site
USD 170,000 - 210,000
Health coverage
Equity
401k matching
+8
Senior InfiniBand & UFM Network Engineer (Hybrid/Remote)
Senior InfiniBand & UFM Network Engineer (Hybrid/Remote)

lightningai • United States

Hybrid
USD 170,000 - 210,000
Health insurance
Unlimited PTO
401(k) matching
+4
Senior Network Engineer - InfiniBand / UFM
Senior Network Engineer - InfiniBand / UFM

lightningai • New York (NY)

On-site
USD 150,000 - 210,000
Senior InfiniBand & GPU Network Engineer
Senior InfiniBand & GPU Network Engineer

ChronoScale Corporation • Palo Alto (CA)

On-site
USD 163,000 - 187,000
Senior AI/HPC Solutions Architect — InfiniBand Deployments
Senior AI/HPC Solutions Architect — InfiniBand Deployments

NVIDIA Corporation • Santa Clara (CA)

Remote
USD 148,000 - 235,750
Senior HPC InfiniBand Network Engineer for AI Infra
Senior HPC InfiniBand Network Engineer for AI Infra

Mirantis • United States

Remote
USD 140,000 - 190,000
AI infrastructure
NVIDIA GPUs
Kubernetes platforms
Senior AI/HPC Solutions Architect - InfiniBand Deployments
Senior AI/HPC Solutions Architect - InfiniBand Deployments

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 148,000 - 236,000
Equity
Benefits
Senior Network Engineer - InfiniBand / UFM
Senior Network Engineer - InfiniBand / UFM

lightningai • United States

Hybrid
USD 170,000 - 210,000
Health insurance
Unlimited PTO
401(k) matching
+4
Senior HPC Support Engineer – InfiniBand/NVLink Expert
Senior HPC Support Engineer – InfiniBand/NVLink Expert

NVIDIA Corporation • Santa Clara (CA)

On-site
USD 120,000 - 230,000
Equity
Benefits package
Senior Network Engineer - InfiniBand / UFM
Senior Network Engineer - InfiniBand / UFM

Lightning AI • San Francisco (CA), Seattle (WA), New York (NY)

On-site
USD 170,000 - 210,000
Health coverage
Equity
401k matching
+8