Multi-Node GPU Testing Engineer for AI Infra

Crusoe; San Francisco (CA); On-site; USD 172,500 - 210,000; Full time

Benefits offered by this job

Competitive compensation

Equity packages

Health insurance

HSA contributions

Parental leave

Disability insurance

Tuition reimbursement

Mental health support

Commuter benefits

Cell phone stipend

401(k) match

Volunteer time off

Travel insurance

Job summary

Crusoe in San Francisco and Sunnyvale is seeking an Automated Testing Engineer to validate large-scale GPU clusters and maintain the automated testing framework for high-performance AI workloads.

You will design multi-node validation tests, develop Python/Go automation, and verify interconnects (NVLink, InfiniBand, RoCE) while ensuring multi-tenant isolation and robust performance across dense GPU environments.

Qualifications

• 5+ years of experience performing responsibilities independently.
• Experience building automated integration testing for AI Cloud environments.
• Knowledge of Kubernetes, Docker, Terraform, and Postgres in production.
• CI/CD pipelines and Gitlab tooling for stable releases across datacenters.
• Python and/or Bash scripting for cluster-wide test automation.
• Familiarity with NVIDIA CUDA/NCCL and AMD ROCm stacks in multi-node setups.
• Strong understanding of RDMA, RoCE, InfiniBand in virtualized environments.
• Knowledge of Linux kernel internals (PCIe, VFIO, HugePages, IOMMU).

Responsibilities

• Build CI/CD platforms to test and deploy low-level systems and applications.
• Design and run large-scale multi-node validation tests to ensure linear scaling and stability.
• Develop automation frameworks in Python or Go for provisioning and stress-testing clusters.
• Validate high-speed interconnects within virtualized environments for low latency and high bandwidth.
• Create test suites using nccl-tests and rccl-tests for cross-node performance.
• Investigate CPU and multi-node communication regressions across guest OS, hypervisor, and hardware fabric.
• Develop test suites with fio, stress-ng, and iperf to ensure multi-tenant isolation.

Skills

Automation & Scripting

Distributed GPU Ecosystems

Networking Knowledge

System Internals

CI/CD & Gitlab

Python

Go

Education

Bachelor's or Master’s degree in Computer Science or Electrical Engineering

Tools

Kubernetes

Docker

Terraform

Postgres

Gitlab

Job description

Crusoe in San Francisco and Sunnyvale is seeking an Automated Testing Engineer to validate large-scale GPU clusters and maintain the automated testing framework for high-performance AI workloads.

You will design multi-node validation tests, develop Python/Go automation, and verify interconnects (NVLink, InfiniBand, RoCE) while ensuring multi-tenant isolation and robust performance across dense GPU environments.