Remote AI Performance Engineer - Training & Inference

Cornelis Networks, Inc.; Oregon; Hybrid; USD 120,000 - 190,000; Full time

Job summary

Cornelis Networks, Inc. is hiring an AI Performance Engineer to optimize training and multi-node inference across Omni-Path, Ethernet, and InfiniBand in GPU/CPU clusters. Collaborate with architecture, firmware, and software teams to turn lab results into field-proven improvements.

You will benchmark OpenAI/ML workloads, design efficient serving topologies, and visualize performance using Nsight, VTune, and Prometheus/Grafana. Remote work options available within the United States.

Qualifications

• BS in CS/EE/CE/Math or related (MS preferred) and 2-3+ years running AI/ML at cluster scale.
• Proven ability to set up, run, and analyze AI benchmarks; deep intuition for message passing, collectives, scaling efficiency, and bottleneck hunting for both training and inference.
• Hands-on with distributed training beyond single-GPU (DP/TP/PP, ZeRO, FSDP, sharded optimizers) and distributed inference architectures (replicated vs sharded, tensor/KV parallel, MoE).
• Practical experience across AI stacks & comms: PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, llama.cpp
• Comfortable with compilers(GCC/LLVM/Intel/OneAPI) and MPIstacks; Python + shell power user.
• Familiarity with network architectures (Omni-Path/OPA, InfiniBand, Ethernet/RDMA/RoCE) and Linuxsystems at the performance-tuning level, including NIC offloads, CQ moderation, pacing, ECN.
• Excellent written and verbal communication—turn measurements intocustomer persuasion.

Responsibilities

• Measure & optimize end-to-end performance for distributed AI workloads (training + multi-node inference) across CPU & GPU clusters and fabrics (Omni-Path, Ethernet, InfiniBand).
• Benchmark, characterize, and tune workloads and benchmarks (e.g., Llama, Qwen) on current/future compute, storage, and network hardware; frameworks include Megatron-LM, TorchTitan, Composer, vLLM, llama.cpp.
• Design and optimize distributed serving topologies (sharded/replicated, tensor/pipe parallel, MoE), batching, KV-cache sharding/offload (CPU/NVMe) & prefix caching.
• Optimize training & inferencing: Validate RDMA/GPUDirect RDMA, congestion control, and collective/point-to-point tradeoffs during inference.
• Build proof points comparing Cornelis Omni-Path to competing interconnects; translate data into narratives for sales/marketing and customers.
• Instrument and visualize performance (Nsight Systems, VTune, perf, app timers) plus Prometheus/Grafana visuals.
• Evangelize best practices through briefs, READMEs, blogs, articles, and peer-reviewed publications.

Skills

AI benchmarks

Distributed training

MPI

PyTorch

DeepSpeed

Megatron-LM

NCCL

Python

Education

Bachelor's degree in CS/EE/CE/Math

MS preferred

Tools

Nsight Systems

VTune

Prometheus Grafana

Nsight Compute

rocprof/roctracer

GCC/LLVM/OneAPI

Job description

Cornelis Networks, Inc. is hiring an AI Performance Engineer to optimize training and multi-node inference across Omni-Path, Ethernet, and InfiniBand in GPU/CPU clusters. Collaborate with architecture, firmware, and software teams to turn lab results into field-proven improvements.

You will benchmark OpenAI/ML workloads, design efficient serving topologies, and visualize performance using Nsight, VTune, and Prometheus/Grafana. Remote work options available within the United States.