Senior AI Cluster Performance & Validation Engineer

Nscale; New York, San Francisco, Seattle (NY, CA, WA); On-site; USD 180,000 - 240,000; Full time

Job summary

Nscale seeks a Principal Infrastructure Engineer to own AI cluster performance and validation across thousands of GPUs. You’ll define healthy-at-scale criteria, set the architecture, and partner with senior leaders to production-readiness goals.

You’ll run real AI workloads at scale, diagnose performance regressions, design scalable validation systems, and optimize fabric, storage, and schedulers to deliver reliable, cost-efficient AI clusters.

Qualifications

• Education: Bachelor's or higher in Computer Science, Engineering or related field.
• Experience: 10+ years in large-scale compute infrastructure with cross-team technical direction.
• AI Workload Expertise: Hands-on experience with real AI compute jobs at scale and distributed training strategies.
• Cluster Validation: Experience validating and accepting large clusters (thousands of GPUs) for performance and reliability.
• Performance Debugging: Ability to diagnose distributed performance problems using NCCL, Nsight, PyTorch Profiler, perf, etc.
• Networking: Deep understanding of InfiniBand/RoCE, RDMA, GPUDirect and related networking.
• Systems & Programming: Strong Linux, Python, C/C++ or Go, and IaC tooling.
• Schedulers: Experience with SLURM and/or Kubernetes at scale.

Responsibilities

• Define healthy-at-scale criteria and production readiness for thousands of GPUs.
• Run real AI workloads to validate cluster behavior under genuine load.
• Diagnose large-scale failures and performance regressions across the full stack.
• Design and build validation and burn-in systems for scalable qualification.
• Drive end-to-end cluster optimization of fabric, storage, and schedulers.
• Collaborate with cross-functional teams to translate needs into durable solutions.
• Build production-grade Python tooling for triage and telemetry.
• Establish engineering standards for reliability and observability.

Skills

AI workloads

GPU clusters

PyTorch

Megatron-LM

DeepSpeed

SLURM

Kubernetes

Python

C/C++

Education

Bachelor's degree or higher in CS/CE or equivalent

Master's degree or PhD (preferred)

Tools

NVIDIA Nsight

Perf

Prometheus

Grafana

OpenTelemetry

Terraform

Ansible

Job description

Nscale seeks a Principal Infrastructure Engineer to own AI cluster performance and validation across thousands of GPUs. You’ll define healthy-at-scale criteria, set the architecture, and partner with senior leaders to production-readiness goals.

You’ll run real AI workloads at scale, diagnose performance regressions, design scalable validation systems, and optimize fabric, storage, and schedulers to deliver reliable, cost-efficient AI clusters.