Senior AI Cluster Performance Engineer

020 Cisco Systems, Inc.; Milpitas (CA); On-site; USD 168,000 - 245,000; Full time

Job summary

Cisco Systems, Inc. is seeking a Senior Software Engineer to debug, profile, and optimize the AI cluster networking and GPU performance stack across boot-time to kernel level, SmartNIC/DPU, and NCCL-based communications.

You will investigate kernel-level failures, tune RDMA and NCCL, and read hardware traces to identify end-to-end bottlenecks while collaborating with hardware and software teams to drive performance improvements.

Qualifications

• Bachelor's degree + 7 years of related experience, or Master's degree + 4 years, or PhD + 1 year, or equivalent related work experience.
• 4+ years of experience with Linux kernel, networking, GPU, and distributed-systems debugging.
• Experience writing code in C/C++/Python and experience with user-mode and kernel-mode drivers.
• Experience with Linux network stack, RDMA/RoCEv2, NCCL, PCIe, and performance profiling.
• Experience reading packet captures, traces, counters, logs, and reproducible benchmarks.

Responsibilities

• Investigate kernel-level failures, boot-time anomalies, PCIe link issues, driver behavior, and network latency.
• Diagnose and tune RDMA/RoCEv2, NCCL, SmartNIC/DPU, NIC queues, congestion, throughput, and collective communications.
• Read and correlate unified hardware traces across host CPUs, PCIe switches, SmartNICs, and GPU compute streams.
• Capture and analyze Wireshark packets, system logs, NVIDIA support logs, traces, profiles, and observability data.
• Partner with hardware and software teams to eliminate end-to-end bottlenecks.

Skills

C/C++/Python

Linux kernel

Networking

Distributed debugging

RDMA/RoCEv2

NCCL

PCIe

Education

Bachelor's degree

Master's degree

PhD

Tools

Nsight profiling tools

Kernel tracing

Observability platforms

Wireshark

Job description

Cisco Systems, Inc. is seeking a Senior Software Engineer to debug, profile, and optimize the AI cluster networking and GPU performance stack across boot-time to kernel level, SmartNIC/DPU, and NCCL-based communications.

You will investigate kernel-level failures, tune RDMA and NCCL, and read hardware traces to identify end-to-end bottlenecks while collaborating with hardware and software teams to drive performance improvements.