Principal Infrastructure Engineer, AI Cluster Performance & Validation

nscaleoperationsukltd

Georgia

Presencial

USD 180.000 - 240.000

Jornada completa

14 días+

Una candidatura completa en un minuto

Currículum y carta de presentación personalizados, listos para enviar.

Supera los filtros ATS

Descripción de la vacante

nscaleoperationsukltd is seeking a Principal Infrastructure Engineer, AI Cluster Performance & Validation, to own architectures and pipelines for multi-thousand-GPU AI clusters. You will run real AI workloads, diagnose failures, and drive end-to-end performance optimizations across the stack.

You will lead validation efforts, build scalable testing systems, and collaborate with Infrastructure, Platform, SRE, and customer teams to translate requirements into durable engineering solutions.

Formación

  • 10+ years in large-scale compute infrastructure design, operation, or debugging.
  • Experience leading cross-team technical direction at staff or principal level.
  • Hands-on AI workload experience at scale with distributed training strategies.
  • Strong benchmarking/acceptance for multi-thousand-GPU clusters.
  • Proficient with NCCL/RCCL, HPC benchmarks, and profiling tools.

Responsabilidades

  • Define architecture, roadmaps, and acceptance criteria for production-ready AI clusters.
  • Run real AI workloads to validate cluster behavior under genuine load.
  • Diagnose large-scale failures and performance regressions across the stack.
  • Build validation and burn-in pipelines to qualify nodes and racks at scale.
  • Tune fabric configuration, topology placement, and storage paths for throughput.
  • Collaborate with multiple teams to translate performance needs into durable solutions.
  • Develop production-grade Python tooling for automated triage and regression detection.
  • Establish reliability, observability, and benchmarking standards across services.

Conocimientos

AI workloads
Distributed training
Performance debugging
High-performance networking
Python programming

Educación

Bachelor's or higher in CS/CE or equivalent

Herramientas

NCCL/RCCL
Nsight Systems/Compute
PyTorch Profiler
Perf tooling

Descripción del empleo

Overview

As a Principal Infrastructure Engineer, AI Cluster Performance & Validation, you will be a critical member of the AI Infrastructure Operations team, responsible for ensuring the acceptance, performance, and scalability of our cutting-edge AI and High-Performance Computing (HPC) environments. Leveraging software engineering and testing principles, you will focus on building and maintaining the control plane, tooling, and automation that supports performance and validation testing of large-scale AI clusters. Your work will directly translate into higher system availability, compute optimization and reduced operational costs.

Key Responsibilities
  • Own the technical definition of "healthy at scale." Set the architecture, roadmap, and acceptance criteria by which multi-thousand-GPU clusters are declared production-ready, and establish the performance bar (collective bandwidth, job goodput, model FLOPs utilization) that every cluster must clear before and after customer handover. Establish technology and product direction in collaboration with other tech leads, managers, and senior leadership.
  • Run and instrument real AI workloads as a diagnostic instrument. Stand up and execute distributed training and inference jobs - open-source and customer-representative models - across thousands of accelerators to validate cluster behavior under genuine load rather than synthetic proxies alone, and translate what those runs reveal into fleet-wide fixes.
  • Lead deep diagnosis of large-scale cluster failures and performance regressions , isolating root cause across the full stack: GPU and NIC firmware, PCIe/NVLink topology and NUMA placement, InfiniBand/RoCE fabric health, congestion control and routing, storage and data-loader throughput, scheduler placement, and framework/communication-library behavior. Serve as the final escalation point for the hardest slow-job and stalled-job investigations.
  • Design and build the validation and burn-in systems that qualify nodes, racks, and full pods at scale - NCCL/RCCL collective sweeps, HPL/HPCG and MLPerf-style benchmarks, thermal and power soak tests, straggler and flapping-link detection - and automate them so that qualification is a repeatable pipeline, not a manual campaign.
  • Drive cluster optimization end to end , tuning fabric configuration (adaptive routing, QoS and congestion control, SHARP in-network reduction, rail and topology-aware placement), collective communication libraries and algorithm selection, GPUDirect RDMA and storage paths, and host-level settings (huge pages, IRQ affinity, CPU governors, MIG and driver configuration) to convert raw hardware into delivered throughput.
  • Partner with Infrastructure, Platform, SRE, and customer-facing teams to translate operational and customer performance needs into durable engineering solutions, and to feed diagnostic signal back into provisioning, remediation, and capacity workflows.
  • Build production-grade Python systems and performance tooling for automated triage, telemetry correlation, and regression detection, leveraging AI tools to accelerate delivery. Assess impact to the team's software and validation stack from new hardware product programs, and explore AI-driven process improvement and automation.
  • Establish engineering standards for reliability, observability, benchmarking methodology, and operational excellence across all services, and raise the diagnostic capability of the wider organization through mentorship, runbooks, and post-incident technical write-ups.
Required Qualifications
  • Education: Bachelor's or higher degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.
  • Experience: 10+ years of relevant experience building, operating, or debugging large-scale compute infrastructure, including significant time at staff or principal level owning cross-team technical direction.
  • AI Workload Expertise: Hands-on experience running real AI compute jobs at scale - pre-training, fine-tuning, or large-scale inference of open-source or proprietary models - including practical familiarity with distributed training strategies (data, tensor, pipeline, and expert parallelism) and frameworks such as PyTorch, Megatron-LM, DeepSpeed, or equivalent.
  • Cluster Validation: Demonstrated experience validating and accepting large clusters (thousands of GPUs) for performance and reliability, with a working command of benchmark methodology and the ability to defend a number to both engineers and customers.
  • Performance Debugging: Proven ability to diagnose distributed performance problems - stragglers, collective stalls, link flaps, thermal throttling, silent data corruption, ECC and Xid errors, noisy-neighbor and storage-bound bottlenecks - using tools such as NCCL debug tracing, Nsight Systems/Compute, PyTorch Profiler, perf, and fabric telemetry.
  • Networking: Deep understanding of high-performance fabrics - InfiniBand and/or RoCEv2, RDMA, GPUDirect, adaptive routing, co
Consigue la evaluación confidencial y gratuita de tu currículum.

o arrastra y suelta tu archivo aquí

Similar jobs

Puestos de trabajo similares que vale la pena comparar

Principal Infrastructure Engineer, AI Cluster Performance & Validation
Principal Infrastructure Engineer, AI Cluster Performance & Validation

nscaleoperationsukltd • EE. UU.

A distancia
USD 180.000 - 240.000
Principal Infrastructure Engineer, AI Cluster Performance & Validation
Principal Infrastructure Engineer, AI Cluster Performance & Validation

Nscale • Houston (TX), New York (NY), San Francisco (CA), Seattle (WA)

Presencial
USD 180.000 - 240.000
Senior Manager, AI and HPC Cluster Engineering Enablement
Senior Manager, AI and HPC Cluster Engineering Enablement

NVIDIA Corporation • EE. UU.

Presencial
USD 180.000 - 260.000
Senior Staff Automation Engineer
Senior Staff Automation Engineer

Crusoe Energy Systems • Seattle (WA)

Presencial
USD 170.000 - 260.000
Health & wellbeing
Paid time off
401(k) match
+1
Principal Engineer - AI & HPC Solutions Architect
Principal Engineer - AI & HPC Solutions Architect

dell • EE. UU.

A distancia
USD 180.000 - 240.000
Senior Automation Engineer (Compute)
Senior Automation Engineer (Compute)

Crusoe Energy Systems • Sunnyvale (CA)

Presencial
USD 180.000 - 240.000
Health benefits
Paid time off
401(k) match
+1
Lead HPC Systems Engineer
Lead HPC Systems Engineer

EPAM Systems Inc • EE. UU.

A distancia
USD 120.000 - 180.000
AI Infrastructure Lead
AI Infrastructure Lead

Blue Signal LLC. • California

Presencial
USD 180.000 - 280.000
Equity participation
Leadership exposure
Ownership of infrastructure
AI Kernel / Cluster Engineer
AI Kernel / Cluster Engineer

Blue Signal LLC. • Santa Clara (CA)

Presencial
USD 150.000 - 210.000
Cluster Design
Cluster Design

Blue Signal LLC. • San Francisco (CA)

Presencial
USD 150.000 - 230.000