Principal AI Network Engineer - Highrise AI

LinuxCareers; United States; Remote; USD 180,000 - 240,000; Full time

Benefits offered by this job

NVIDIA networking tools

Job summary

Highrise AI seeks a principal-level network leader to deploy and operate large-scale GPU clusters. You will bring physical infrastructure online, validate topology and cabling, and provide Tier 3 escalation for complex networking issues.

You will design automation, runbooks, and monitoring standards for NOC/SRE handoff while collaborating with compute engineers to optimize multi-node AI workloads.

Qualifications

• Bachelor's degree in computer science, engineering, networking, or related discipline, or equivalent practical experience.
• 10+ years of hands-on data center or high-performance networking experience with deep troubleshooting and technical leadership.
• Experience deploying and operating large-scale, low-latency networks for AI, HPC, cloud, or similar distributed systems.
• Strong expertise in Ethernet and/or InfiniBand fabrics, RDMA/RoCE, routing and switching.
• Proficiency with Linux networking and network automation using Python, Ansible, or equivalent tooling.
• Expert-level problem solving across physical, link, transport, and application-facing network performance layers.

Responsibilities

• Lead network fabric bring-up and commissioning of AI clusters, including switch initialization and RDMA setup.
• Execute fabric qualification testing for link health, latency, and resilience; tune RDMA and QoS.
• Analyze NCCL/distributed workload results with compute engineers to improve multi-node performance.
• Provide Tier 3 escalation for production incidents involving fabric instability and network performance.
• Develop network automation, configuration management procedures, and runbooks for NOC/SRE handoff.

Skills

10+ years networking experience

Ethernet/InfiniBand fabrics

RDMA/RoCE

Linux networking

Python

Ansible

Troubleshooting

Education

Bachelor's degree in CS/engineering/networking

Tools

NVIDIA networking tools

Job description

Highrise AI is a GPU-native infrastructure platform for modern AI workloads. This principal-level role leads large-scale GPU cluster network deployment, configuration, and operations—bringing physical installation through production readiness, with ongoing Tier 3 technical escalation for complex networking issues.

What You’ll Do

• Lead network fabric bring-up and commissioning of AI clusters, including switch initialization, high-speed Ethernet/InfiniBand configuration, RDMA/RoCE setup, and validation of topology, routing, redundancy, and cabling


• Execute fabric qualification testing for link health, bandwidth, latency, congestion, and resilience; troubleshoot and tune RDMA transport, QoS, PFC/ECN, and switch/NIC settings


• Analyze NCCL and distributed workload results with compute engineers to isolate communication bottlenecks and improve multi-node performance


• Provide Tier 3 technical escalation for production incidents involving fabric instability, host connectivity, and network performance; lead deep troubleshooting and root cause analysis across switches, NICs, optics, and cabling


• Develop network automation, configuration management procedures, diagnostic automation, monitoring standards, and runbooks for NOC and SRE handoff

What You Need

• Bachelor's degree in computer science, engineering, networking, or related discipline, or equivalent practical experience


• 10+ years of hands-on data center or high-performance networking experience with deep troubleshooting and technical leadership


• Demonstrated success deploying and operating large-scale, low-latency networks for AI, HPC, cloud, or similar distributed systems


• Strong expertise in Ethernet and/or InfiniBand fabrics, RDMA/RoCE, routing and switching, and multi-tier data center network architectures


• Proficiency with Linux networking and network automation using Python, Ansible, or equivalent tooling


• Expert-level problem solving across physical, link, transport, and application-facing network performance layers

Nice to Have

• Experience with high-speed 200/400/800G networking, NICs, optical interconnects, performance testing, and network telemetry


• Familiarity with NVIDIA networking and fabric-management tools