Senior InfiniBand Network Engineer - UFM, HPC Fabric | Hybrid

Lightning AI; New York (NY); Hybrid; USD 170,000 - 210,000; Full time

Benefits offered by this job

Comprehensive Health Coverage

Meaningful Equity

401(k) matching

Unlimited PTO

Winter Break

Parental Leave

Learning allowance

Wellness stipend

Sabbatical program

Hybrid work model

In-Office Meals

Job summary

Lightning AI in New York seeks a Senior Network Engineer to design, deploy, and maintain high-performance InfiniBand fabrics for AI training and inference across GPU clusters. You will work with UFM, NCCL, and modern data center tech to ensure scalable, reliable networking.

The role emphasizes automation (Python, Ansible) and collaboration with AI platform and storage teams. Competitive compensation, equity, and benefits are offered.

Qualifications

• 7+ years of data center networking experience.
• 3+ years supporting NVIDIA InfiniBand environments.
• Hands-on experience with NVIDIA UFM Enterprise.
• Experience deploying and operating Quantum and Quantum-2 InfiniBand switches.
• Strong Linux administration experience (Ubuntu).
• Automation with Python and Ansible.
• Experience with BGP, EVPN, VXLAN and spine-leaf architectures.

Responsibilities

• Design, deploy, and maintain large-scale NVIDIA InfiniBand fabrics supporting AI/ML GPU clusters.
• Deploy and administer NVIDIA Unified Fabric Manager (UFM) Enterprise for monitoring, provisioning, telemetry, and fabric health.
• Configure and optimize NVIDIA Quantum and Quantum-2 InfiniBand switches.
• Troubleshoot fabric performance issues impacting NCCL, MPI, GPUDirect RDMA, and AI training jobs.
• Implement and validate fat-tree, Dragonfly+, Clos, and spine-leaf network architectures.
• Perform firmware lifecycle management for InfiniBand switches, adapters (HCAs), and UFM infrastructure.
• Optimize congestion control, adaptive routing, QoS, and traffic engineering for high-performance GPU communication.
• Automate network provisioning using Python, Ansible, Git, REST APIs, and IaC.
• Monitor network health using UFM telemetry, Prometheus, Grafana, and other observability platforms.
• Support high availability, maintenance windows, incident response, root cause analysis, and capacity planning.
• Participate in architecture reviews and define networking standards for AI infrastructure.

Skills

InfiniBand networking

NVIDIA UFM

NVIDIA switches

Linux administration

Python automation

Ansible automation

NCCL MPI

Observability tooling

Tools

NVIDIA Quantum

NVIDIA Quantum-2

ibdiagnet

tcpdump

Wireshark

Prometheus

Grafana

UFM Enterprise

Terraform

Job description

Lightning AI in New York seeks a Senior Network Engineer to design, deploy, and maintain high-performance InfiniBand fabrics for AI training and inference across GPU clusters. You will work with UFM, NCCL, and modern data center tech to ensure scalable, reliable networking.

The role emphasizes automation (Python, Ansible) and collaboration with AI platform and storage teams. Competitive compensation, equity, and benefits are offered.