Senior InfiniBand & UFM Network Engineer (Hybrid/Remote)

lightningai

United States

Hybrid

USD 170,000 - 210,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
Unlimited PTO
401(k) matching
Sabbatical after 4 years
Learning & development allowance
Wellness stipend
Work-from-home stipend

Job summary

LightningAI seeks a senior network engineer to design, deploy, and operate the high‑performance InfiniBand fabric that underpins large‑scale AI training and inference workloads. You will build and optimize GPU‑dense clusters and automate provisioning with Python and Ansible, while monitoring health with UFM, Prometheus, and Grafana.

You’ll lead capacity planning, incident response, and topology validation for fat-tree, Dragonfly+, and spine‑leaf designs, ensuring deterministic performance at

Qualifications

  • 7+ years of experience operating complex network infrastructure at scale.
  • Deep expertise with InfiniBand networking, NVIDIA UFM, and large-scale GPU deployments.
  • Strong Linux networking fundamentals and hands-on troubleshooting of distributed workloads.
  • Proficiency with NCCL, RoCEv2, GPUDirect RDMA, and high-performance communication patterns.

Responsibilities

  • Design, deploy, and maintain large-scale NVIDIA InfiniBand fabrics powering AI/ML GPU clusters.
  • Administer NVIDIA Unified Fabric Manager (UFM) Enterprise for provisioning, telemetry, monitoring, and fabric health.
  • Configure and optimize NVIDIA Quantum and Quantum-2 InfiniBand switches, including firmware lifecycle management.
  • Troubleshoot fabric-level performance issues affecting NCCL, MPI, GPUDirect RDMA, and distributed training jobs.
  • Implement and validate fat-tree, Dragonfly+, Clos, and spine-leaf topologies for high-throughput AI environments.
  • Automate network provisioning through Python, Ansible, Git, REST APIs, and Infrastructure-as-Code practices.
  • Monitor infrastructure with UFM telemetry, Prometheus, and Grafana; lead incident response and capacity planning.

Skills

InfiniBand networking
NVIDIA UFM
GPU deployments
Linux networking
NCCL & GPUDirect
Traffic engineering
Automation tooling (Python, Ansible)
Git
Observability platforms

Tools

Ansible
Terraform
Git

Job description

LightningAI seeks a senior network engineer to design, deploy, and operate the high‑performance InfiniBand fabric that underpins large‑scale AI training and inference workloads. You will build and optimize GPU‑dense clusters and automate provisioning with Python and Ansible, while monitoring health with UFM, Prometheus, and Grafana.

You’ll lead capacity planning, incident response, and topology validation for fat-tree, Dragonfly+, and spine‑leaf designs, ensuring deterministic performance at

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior InfiniBand Network Architect: UFM & HPC Infra
Senior InfiniBand Network Architect: UFM & HPC Infra

lightningai • New York (NY)

On-site
USD 150,000 - 210,000
Senior InfiniBand Network Engineer (AI Clusters)
Senior InfiniBand Network Engineer (AI Clusters)

Lightning AI • San Francisco (CA), Seattle (WA), New York (NY)

On-site
USD 170,000 - 210,000
Health coverage
Equity
401k matching
+8
Senior InfiniBand & GPU Network Engineer
Senior InfiniBand & GPU Network Engineer

ChronoScale Corporation • Palo Alto (CA)

On-site
USD 163,000 - 187,000
Senior HPC InfiniBand Network Engineer for AI Infra
Senior HPC InfiniBand Network Engineer for AI Infra

Mirantis • United States

Remote
USD 140,000 - 190,000
AI infrastructure
NVIDIA GPUs
Kubernetes platforms
GPU Network Engineer: InfiniBand Fabric Architect (Remote)
GPU Network Engineer: InfiniBand Fabric Architect (Remote)

Andromeda • United States

Hybrid
USD 180,000 - 240,000
Healthcare
Dental
Vision
+2
Senior Network Engineer - InfiniBand / UFM
Senior Network Engineer - InfiniBand / UFM

lightningai • New York (NY)

On-site
USD 150,000 - 210,000
Senior InfiniBand HPC Networking Architect – Remote
Senior InfiniBand HPC Networking Architect – Remote

Energy Jobline ZR • Miami (FL)

Hybrid
USD 185,000 - 200,000
Senior Network Engineer - InfiniBand / UFM
Senior Network Engineer - InfiniBand / UFM

lightningai • United States

Hybrid
USD 170,000 - 210,000
Health insurance
Unlimited PTO
401(k) matching
+4
Senior Network Engineer - InfiniBand / UFM
Senior Network Engineer - InfiniBand / UFM

Lightning AI • San Francisco (CA), Seattle (WA), New York (NY)

On-site
USD 170,000 - 210,000
Health coverage
Equity
401k matching
+8
Senior Network Engineer (InfiniBand & HPC Architecture) in Miami
Senior Network Engineer (InfiniBand & HPC Architecture) in Miami

Energy Jobline ZR • Miami (FL)

Hybrid
USD 185,000 - 200,000