Principal AI Infra Architect – Scale & Validation

nscaleoperationsukltd

Georgia

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

nscaleoperationsukltd is seeking a Principal Infrastructure Engineer, AI Cluster Performance & Validation, to own architectures and pipelines for multi-thousand-GPU AI clusters. You will run real AI workloads, diagnose failures, and drive end-to-end performance optimizations across the stack.

You will lead validation efforts, build scalable testing systems, and collaborate with Infrastructure, Platform, SRE, and customer teams to translate requirements into durable engineering solutions.

Qualifications

  • 10+ years in large-scale compute infrastructure design, operation, or debugging.
  • Experience leading cross-team technical direction at staff or principal level.
  • Hands-on AI workload experience at scale with distributed training strategies.
  • Strong benchmarking/acceptance for multi-thousand-GPU clusters.
  • Proficient with NCCL/RCCL, HPC benchmarks, and profiling tools.

Responsibilities

  • Define architecture, roadmaps, and acceptance criteria for production-ready AI clusters.
  • Run real AI workloads to validate cluster behavior under genuine load.
  • Diagnose large-scale failures and performance regressions across the stack.
  • Build validation and burn-in pipelines to qualify nodes and racks at scale.
  • Tune fabric configuration, topology placement, and storage paths for throughput.
  • Collaborate with multiple teams to translate performance needs into durable solutions.
  • Develop production-grade Python tooling for automated triage and regression detection.
  • Establish reliability, observability, and benchmarking standards across services.

Skills

AI workloads
Distributed training
Performance debugging
High-performance networking
Python programming

Education

Bachelor's or higher in CS/CE or equivalent

Tools

NCCL/RCCL
Nsight Systems/Compute
PyTorch Profiler
Perf tooling

Job description

nscaleoperationsukltd is seeking a Principal Infrastructure Engineer, AI Cluster Performance & Validation, to own architectures and pipelines for multi-thousand-GPU AI clusters. You will run real AI workloads, diagnose failures, and drive end-to-end performance optimizations across the stack.

You will lead validation efforts, build scalable testing systems, and collaborate with Infrastructure, Platform, SRE, and customer teams to translate requirements into durable engineering solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Principal Infra Engineer — AI & HPC Scale
Principal Infra Engineer — AI & HPC Scale

nscaleoperationsukltd • United States

Remote
USD 180,000 - 240,000
Principal Infrastructure Engineer, AI Cluster Performance & Validation
Principal Infrastructure Engineer, AI Cluster Performance & Validation

nscaleoperationsukltd • Georgia

On-site
USD 180,000 - 240,000
Principal AI Systems Engineer, GPU Compute Infra
Principal AI Systems Engineer, GPU Compute Infra

Highrise AI • United States

Hybrid
USD 180,000 - 280,000
Senior GPU AI Infrastructure Architect - Equity Eligible
Senior GPU AI Infrastructure Architect - Equity Eligible

nscaleoperationsukltd • United States

Hybrid
USD 180,000 - 260,000
Equity
Base pay + equity
Annual compensation reviews
+1
Principal Infrastructure Engineer, AI Cluster Performance & Validation
Principal Infrastructure Engineer, AI Cluster Performance & Validation

nscaleoperationsukltd • United States

Remote
USD 180,000 - 240,000
Principal Infrastructure Engineer, AI Cluster Performance & Validation
Principal Infrastructure Engineer, AI Cluster Performance & Validation

Nscale • Houston (TX), New York (NY), San Francisco (CA), Seattle (WA)

On-site
USD 180,000 - 240,000
Senior AI Network Architect — High-Performance GPU Clusters
Senior AI Network Architect — High-Performance GPU Clusters

Highrise AI • United States

Hybrid
USD 180,000 - 230,000
Senior AI Compute Infrastructure Architect
Senior AI Compute Infrastructure Architect

ARM • Seattle (WA)

Hybrid
USD 263,000 - 355,000
Relocation package
Visa sponsorship
Principal Software Engineer – AI-Driven Infra Orchestration
Principal Software Engineer – AI-Driven Infra Orchestration

nscaleoperationsukltd • United States

Remote
USD 240,000 - 400,000
Medical insurance
Dental insurance
Vision coverage
+3
Principal AI Inference Architect – Post-Training
Principal AI Inference Architect – Post-Training

nscaleoperationsukltd • Georgia

On-site
USD 260,000 - 360,000