Principal AI Cluster Performance & Validation Architect

Socket.dev; Houston (TX); On-site; USD 200,000 - 260,000; Full time

Job summary

Socket.dev is seeking a Principal Infrastructure Engineer for AI Cluster Performance & Validation to own the healthy-at-scale criteria, define architecture for multi-thousand-GPU clusters, and steer production readiness with strong metrics.

You will run real AI workloads, diagnose large-scale failures, design scalable validation systems, optimize fabric, storage, and schedulers, and collaborate across teams to raise reliability and observability.

Qualifications

• Bachelor's or higher degree in CS/CE or related field.
• 10+ years of experience building/operating large-scale compute infrastructure.
• Hands-on AI workload experience at scale with distributed training strategies.
• Experience validating and accepting large GPU clusters for performance and reliability.
• Proficient in NCCL/RCCL, Nsight, PyTorch Profiler, and related tools.
• Deep Linux systems expertise and production Python, plus C/C++ or Go.
• Experience with SLURM and/or Kubernetes at scale.

Responsibilities

• Define healthy-at-scale criteria and production readiness for multi-thousand-GPU clusters.
• Run real AI workloads to validate cluster behavior under genuine load.
• Diagnose large-scale failures and performance regressions across the full stack.
• Design and build validation and burn-in systems for qualification at scale.
• Optimize clusters end-to-end: fabric configs, topology, storage, and schedulers.
• Collaborate with cross-functional teams to translate needs into engineering solutions.
• Develop production-grade Python tooling for telemetry and triage.
• Establish reliability and benchmarking standards across services.

Skills

AI workloads

Cluster validation

Performance debugging

Networking

Systems programming

Schedulers

Python

C/C++

Education

Bachelor's degree in CS/CE or related field

Tools

Ansible

Terraform

NCCL

Nsight Systems

PyTorch

Kubernetes

Docker

DCGM

Job description

Socket.dev is seeking a Principal Infrastructure Engineer for AI Cluster Performance & Validation to own the healthy-at-scale criteria, define architecture for multi-thousand-GPU clusters, and steer production readiness with strong metrics.

You will run real AI workloads, diagnose large-scale failures, design scalable validation systems, optimize fabric, storage, and schedulers, and collaborate across teams to raise reliability and observability.