AI Performance Engineer

Cornelis Networks, Inc.

United States

Remote

USD 120,000 - 180,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Cornelis Networks is hiring an AI Performance Engineer to optimize training and multi-node inference across CPU/GPU clusters and diverse fabrics. You will work with architecture, firmware, software, and lighthouse customers to turn lab results into field-ready wins.

Responsibilities include measuring end-to-end performance, benchmarking ML workloads, and designing scalable serving topologies. Remote within the United States; strong emphasis on communication and collaboration.

Qualifications

  • 2–3+ years running AI/ML at cluster scale.
  • Ability to set up, run, and analyze AI benchmarks; deep intuition for message passing and scaling bottlenecks.
  • Hands-on with distributed training (DP/TP/PP, ZeRO, FSDP) and distributed inference architectures.
  • Experience with PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, llama.cpp; comfortable with GCC/LLVM and MPI; Python.
  • Familiarity with Omni-Path/OPA, InfiniBand, Ethernet/RDMA; Linux systems at perf-tuning level.
  • Excellent written and verbal communication.

Responsibilities

  • Measure & optimize end-to-end performance for AI workloads across CPU & GPU clusters.
  • Benchmark, characterize, and tune workloads on current/future hardware with relevant ML stacks.
  • Design and optimize distributed serving topologies and batching strategies.
  • Validate RDMA, congestion control, and tradeoffs during inference.
  • Translate lab results into field-ready narratives for customers.

Skills

AI benchmarks
Distributed training
Performance tuning
PyTorch/DeepSpeed
MPI/NCCL
Linux perf tuning
Communication

Education

BS in CS/EE/CE/Math
MS preferred

Tools

Omni-Path/OPA
InfiniBand
Lustre/NVMe-oF

Job description

Cornelis Networks delivers the world’s highest performance scale-out networking solutions for AI and HPC datacenters. Our differentiated architecture integrates hardware, software, and system technologies to maximize the efficiency of GPU, CPU, and accelerator-based clusters at any scale. We’re committed to innovation, performance, and scalability—helping customers push the boundaries of what’s possible. We are a fast-growing, forward-thinking team of architects, engineers, and business professionals with a proven track record of building successful products and companies. As a global organization, our team spans multiple U.S. states and six countries, and we continue to expand with exceptional talent in onsite, hybrid, and fully remote roles.

We’re seeking an AI Performance Engineer. Join us to optimize training and multi-node inference across next-gen networking silicon and systems—adapters, switches, and the software stack that ties it all together. You’ll partner with architecture, firmware, software, and lighthouse customers to turn lab results into field-proven wins.

Key Responsibilities
  • Measure & optimize end-to-end performance for distributed AI workloads (training + multi-node inference) across CPU & GPU clusters and diverse fabrics (Omni-Path, Ethernet, InfiniBand).
  • Benchmark, characterize, and tune open-source & industry workloads and benchmarks (e.g., Llama, Qwen) on current and future compute, storage, and network hardware, including frameworks such as Megatron-LM, TorchTitan, Composer, vLLM, llama.cpp, etc.
  • Design and optimize distributed serving topologies (sharded/replicated, tensor/pipe parallel, MoE expert placement), continuous/adaptive batching, KV-cache sharding/offload (CPU/NVMe) & prefix caching.
  • Optimize training & inferencing: Validate RDMA/GPUDirect RDMA, congestion control, and collective/point-to-point tradeoffs during inference.
  • Build crisp proof points that compare Cornelis Omni-Path to competing interconnects; translate data into narratives for sales/marketing and existing customers.
  • Instrument and visualize performance (Nsight Systems, VTune, perf, app timers) plus database visualization (Prometheus/Grafana).
  • Evangelize best practices through briefs, READMEs, blogs, articles, and peer-reviewed publications.
Minimum Qualification
  • BS in CS/EE/CE/Math or related (MS preferred) and 2-3+ years running AI/ML at cluster scale.
  • Proven ability to set up, run, and analyze AI benchmarks; deep intuition for message passing, collectives, scaling efficiency, and bottleneck hunting for both training and inference.
  • Hands-on with distributed training beyond single-GPU (DP/TP/PP, ZeRO, FSDP, sharded optimizers) and distributed inference architectures (replicated vs sharded, tensor/KV parallel, MoE).
  • Practical experience across AI stacks & comms: PyTorch, DeepSpeed, Megatron-LM, RCCL/NCCL, vLLM, llama.cpp Comfortable with compilers (GCC/LLVM/Intel/OneAPI) and MPI stacks; Python + shell power user.
  • Familiarity with network architectures (Omni-Path/OPA, InfiniBand, Ethernet/RDMA/RoCE) and Linux systems at the performance-tuning level, including NIC offloads, CQ moderation, pacing, ECN.
  • Excellent written and verbal communication—turn measurements into customer persuasion.
Preferred Qualifications
  • Cluster scheduler experience (e.g. SLURM, PBS).
  • Hands-on profiling & tracing of GPU/comm paths (Nsight Systems, Nsight Compute, ROCm tools/rocprof/roctracer, VTune, perf, etc.).
  • Background in HPC performance engineering or storage (BeeGFS, Lustre, NVMe-oF).
Location

Remote: This role fully supports remote work for employees residing within the United States. Cornelis is a company that is strongly led by our core values and believes in the responsibility of our people, which is why we fully support remote employees. When safe, we encourage travel for partner and client meetings, conferences, and company in-person collaboration and meetings. Expect travel to be less than 10%.

Cornelis Networks is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Performance Engineer
AI Performance Engineer

Cornelis Networks, Inc. • San Jose (CA)

Remote
USD 140,000 - 200,000
Senior Software Engineer – AI Middleware
Senior Software Engineer – AI Middleware

Cornelis Networks • Austin (TX)

Remote
USD 120,000 - 150,000
Equity, cash, and incentives
Health coverage
401(k) with company match
+2
Senior Software Engineer – AI Middleware
Senior Software Engineer – AI Middleware

Cornelis Networks, Inc. • United States

Remote
USD 140,000 - 190,000
Equity
Cash incentives
Health benefits
+2
Senior Software Engineer – AI/HPC Middleware
Senior Software Engineer – AI/HPC Middleware

Cornelis Networks, Inc. • United States

Remote
USD 180,000 - 280,000
Equity
Health benefits
401(k) match
AI Performance Engineer for Scale-Out AI Clusters
AI Performance Engineer for Scale-Out AI Clusters

Cornelis Networks, Inc. • San Jose (CA)

Remote
USD 140,000 - 200,000
Principal Embedded Firmware Engineer — AI/HPC Networking Silicon
Principal Embedded Firmware Engineer — AI/HPC Networking Silicon

Cornelis Networks, Inc. • United States

Remote
USD 180,000 - 240,000
Equity
Health insurance
401(k) retirement plan
Principal Embedded Firmware Engineer — AI/HPC Networking Silicon
Principal Embedded Firmware Engineer — AI/HPC Networking Silicon

Cornelis Networks, Inc. • Boston (MA)

Remote
USD 150,000 - 210,000
Equity
Health benefits
401(k) with match
+2
AI Systems Performance Engineer
AI Systems Performance Engineer

Broadcom Inc. • San Jose (CA)

On-site
USD 141,300 - 226,000
Medical, dental, and vision plans
401(k) participation with company matching
Employee Stock Purchase Program (ESPP)
+1
Software Program Manager - HPC
Software Program Manager - HPC

Cornelis Networks, Inc. • Austin (TX)

Remote
USD 120,000 - 180,000
Senior AI Networking & Performance Engineer
Senior AI Networking & Performance Engineer

NVIDIA Corporation • Texas

On-site
USD 272,000 - 431,250