Principal Infra Engineer — AI & HPC Scale

nscaleoperationsukltd; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

nscaleoperationsukltd seeks a principal-level infrastructure leader to set the technical bar for production readiness and performance across very large AI and HPC clusters. This role combines architecture, hands-on workload testing, deep failure analysis and tooling to improve reliability and delivered compute capacity.

Responsibilities include defining cluster health criteria, running distributed training/inference at scale, troubleshooting regressions, and mentoring teams through runbooks and

Qualifications

• Bachelor’s degree or equivalent practical experience in a relevant technical field.
• At least 10 years building, operating, or debugging large-scale compute infrastructure with leadership across teams.
• Hands-on experience running large-scale AI training or inference with frameworks such as PyTorch, Megatron-LM, or DeepSpeed.
• Experience qualifying large GPU clusters and defending benchmark methods and results to technical and customer audiences.
• Strong skills diagnosing distributed performance problems with tools such as NCCL tracing, Nsight, PyTorch Profiler, perf, or fabric telemetry.
• Deep knowledge of InfiniBand or RoCE, RDMA, Linux systems behavior, and production Python; experience with C/C++ or Go and SLURM or Kubernetes at scale.
• Nice to have: graduate degree; production deployment of GPU superclusters; familiarity with NVIDIA/AMD accelerator stacks, parallel storage, observability platforms, infrastructure-as-code, and bare-metal management; or published benchmark work.

Responsibilities

• Define cluster health criteria, performance targets, and acceptance processes for clusters containing thousands of GPUs, including before and after customer handover.
• Run distributed training and inference workloads at scale, use results to identify fleet-wide issues, and translate findings into engineering improvements.
• Investigate slow or stalled jobs and cluster regressions across accelerators, firmware, host topology, high-speed networking, storage, schedulers, and communication libraries.
• Build automated qualification and burn-in pipelines for nodes, racks, and clusters, covering benchmarks, collective communication, thermal and power behavior, and link stability.
• Tune networking, communication libraries, storage paths, and host settings to improve usable workload throughput.
• Develop Python tools for triage, telemetry correlation, and regression detection; establish validation and observability practices; and mentor teams through runbooks and incident reviews.
• Work with infrastructure, platform, reliability, and customer-facing teams to incorporate operational needs into provisioning and remediation workflows.

Skills

Distributed compute leadership

Python tooling

Performance debugging

GPU cluster know-how

NCCL tracing / profiling

SLURM / Kubernetes at scale

Frameworks: PyTorch / Megatron-LM / De

Education

Bachelor’s degree or equivalent practical experience in a relevant technical field

Tools

Nsight

PyTorch Profiler

perf

Fabric Telemetry

SLURM

Kubernetes

Job description

nscaleoperationsukltd seeks a principal-level infrastructure leader to set the technical bar for production readiness and performance across very large AI and HPC clusters. This role combines architecture, hands-on workload testing, deep failure analysis and tooling to improve reliability and delivered compute capacity.

Responsibilities include defining cluster health criteria, running distributed training/inference at scale, troubleshooting regressions, and mentoring teams through runbooks and