Principal Infrastructure Engineer, AI Cluster Performance & Validation

nscaleoperationsukltd; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

nscaleoperationsukltd seeks a principal-level infrastructure leader to set the technical bar for production readiness and performance across very large AI and HPC clusters. This role combines architecture, hands-on workload testing, deep failure analysis and tooling to improve reliability and delivered compute capacity.

Responsibilities include defining cluster health criteria, running distributed training/inference at scale, troubleshooting regressions, and mentoring teams through runbooks and

Qualifications

• Bachelor’s degree or equivalent practical experience in a relevant technical field.
• At least 10 years building, operating, or debugging large-scale compute infrastructure with leadership across teams.
• Hands-on experience running large-scale AI training or inference with frameworks such as PyTorch, Megatron-LM, or DeepSpeed.
• Experience qualifying large GPU clusters and defending benchmark methods and results to technical and customer audiences.
• Strong skills diagnosing distributed performance problems with tools such as NCCL tracing, Nsight, PyTorch Profiler, perf, or fabric telemetry.
• Deep knowledge of InfiniBand or RoCE, RDMA, Linux systems behavior, and production Python; experience with C/C++ or Go and SLURM or Kubernetes at scale.
• Nice to have: graduate degree; production deployment of GPU superclusters; familiarity with NVIDIA/AMD accelerator stacks, parallel storage, observability platforms, infrastructure-as-code, and bare-metal management; or published benchmark work.

Responsibilities

• Define cluster health criteria, performance targets, and acceptance processes for clusters containing thousands of GPUs, including before and after customer handover.
• Run distributed training and inference workloads at scale, use results to identify fleet-wide issues, and translate findings into engineering improvements.
• Investigate slow or stalled jobs and cluster regressions across accelerators, firmware, host topology, high-speed networking, storage, schedulers, and communication libraries.
• Build automated qualification and burn-in pipelines for nodes, racks, and clusters, covering benchmarks, collective communication, thermal and power behavior, and link stability.
• Tune networking, communication libraries, storage paths, and host settings to improve usable workload throughput.
• Develop Python tools for triage, telemetry correlation, and regression detection; establish validation and observability practices; and mentor teams through runbooks and incident reviews.
• Work with infrastructure, platform, reliability, and customer-facing teams to incorporate operational needs into provisioning and remediation workflows.

Skills

Distributed compute leadership

Python tooling

Performance debugging

GPU cluster know-how

NCCL tracing / profiling

SLURM / Kubernetes at scale

Frameworks: PyTorch / Megatron-LM / De

Education

Bachelor’s degree or equivalent practical experience in a relevant technical field

Tools

Nsight

PyTorch Profiler

perf

Fabric Telemetry

SLURM

Kubernetes

Job description

Role overview

Set the technical bar for production readiness and performance across very large AI and HPC clusters. This principal-level role combines infrastructure architecture, hands-on workload testing, deep failure analysis, and software tooling to improve reliability and delivered compute capacity.

Responsibilities

• Define cluster health criteria, performance targets, and acceptance processes for clusters containing thousands of GPUs, including before and after customer handover.
• Run distributed training and inference workloads at scale, use their results to identify fleet-wide issues, and turn findings into engineering improvements.
• Investigate slow or stalled jobs and cluster regressions across accelerators, firmware, host topology, high-speed networking, storage, schedulers, and communication libraries.
• Build automated qualification and burn-in pipelines for nodes, racks, and clusters, covering benchmarks, collective communication, thermal and power behavior, and link stability.
• Tune networking, communication libraries, storage paths, and host settings to improve usable workload throughput.
• Develop Python tools for triage, telemetry correlation, and regression detection; establish validation and observability practices; and mentor teams through runbooks and incident reviews.
• Work with infrastructure, platform, reliability, and customer-facing teams to incorporate operational needs into provisioning and remediation workflows.

Requirements

• Bachelor’s degree or equivalent practical experience in a relevant technical field.
• At least 10 years building, operating, or debugging large-scale compute infrastructure, including substantial staff- or principal-level technical leadership across teams.
• Hands-on experience running large-scale AI training or inference, with knowledge of distributed parallelism and frameworks such as PyTorch, Megatron-LM, or DeepSpeed.
• Experience qualifying large GPU clusters and defending benchmark methods and results to technical and customer audiences.
• Strong skills diagnosing distributed performance problems with tools such as NCCL tracing, Nsight, PyTorch Profiler, perf, or fabric telemetry.
• Deep knowledge of InfiniBand or RoCE, RDMA, Linux systems behavior, and production Python; experience with C/C++ or Go and SLURM or Kubernetes at scale.

Nice to have

• A graduate degree; experience bringing a new GPU supercluster into production; familiarity with NVIDIA or AMD accelerator stacks, parallel storage, observability platforms, infrastructure-as-code, and bare-metal management systems; or published benchmark and scaling work.