Principal AI Cluster Performance & Validation Engineer

Devopsroles; Houston (TX); In loco; USD 180.000 - 240.000; Tempo pieno

Descrizione del lavoro

Nscale is seeking a Principal Infrastructure Engineer to lead AI cluster performance and validation. You will define healthy-at-scale criteria, orchestrate real AI workloads, and drive end-to-end optimization across thousands of GPUs in production environments.

You will pair Linux expertise with Python and C/C++ or Go, work with SLURM and Kubernetes, and build automation and telemetry to improve reliability, throughput, and cost efficiency.

Competenze

• Bachelor's or higher in Computer Science, Computer Engineering, or equivalent.
• 10+ years in large-scale compute infrastructure, staff or principal level.
• Hands-on AI compute experience at scale with distributed training strategies.
• Experience validating clusters of thousands of GPUs for performance and reliability.
• Strong debugging of distributed performance with NCCL, Nsight, PyTorch Profiler, and perf.
• Deep understanding of InfiniBand/RoCE, RDMA, GPUDirect, and network topologies.
• Proficiency in Linux systems, Python, and C/C++ or Go; IaC tools like Ansible, Terraform.
• Experience with SLURM and/or Kubernetes at scale.

Mansioni

• Own the definition of healthy at scale and production readiness for multi-thousand-GPU clusters.
• Run real AI workloads to validate cluster behavior under genuine load.
• Diagnose large-scale failures and performance regressions across the stack.
• Design validation and burn-in systems with benchmarks and soak tests.
• Build production-grade performance tooling and telemetry for triage and detection.
• Collaborate with cross-functional teams to translate needs into durable solutions.
• Establish engineering standards for reliability and observability.

Conoscenze

AI workloads

Distributed training

Linux systems

Python

C/C++

Go

Kubernetes

SLURM

Formazione

Bachelor's degree in Computer Science

Bachelor's degree in Computer Engineering

Strumenti

NCCL

Nsight Systems

PyTorch Profiler

perf

Fabric telemetry

Prometheus

Grafana

Descrizione del lavoro

Nscale is seeking a Principal Infrastructure Engineer to lead AI cluster performance and validation. You will define healthy-at-scale criteria, orchestrate real AI workloads, and drive end-to-end optimization across thousands of GPUs in production environments.

You will pair Linux expertise with Python and C/C++ or Go, work with SLURM and Kubernetes, and build automation and telemetry to improve reliability, throughput, and cost efficiency.