Senior Validation TPM for GPU AI Infra Clusters

nscaleoperationsukltd; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Nscale is seeking a Validation TPM to transform customer acceptance criteria into scalable test plans and to drive end-to-end validation of large GPU clusters. You will work between engineering, OEM partners and the customer, delivering results that determine handoff readiness.

Bring 7-10+ years in software validation for infrastructure, strong scripting in Python/Bash, and deep knowledge of GPU cluster performance. Expect travel up to 50% to data center sites as part of the role.

Qualifications

• 7-10+ years in software validation, test engineering or QA for infrastructure or distributed systems, including program or project ownership.
• Hands-on experience building and running test frameworks, and reading and debugging their output.
• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn‑in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, in writing and in the room.
• Ability to travel up to 50% of the time, primarily to data center sites.

Responsibilities

• Turn acceptance criteria into tests: translate each customer's acceptance criteria into a clear test plan, what is run, at what scale, and with what pass thresholds.
• Agree the plan with the customer and engineering before testing starts.
• Map every acceptance criterion to the test result that proves it, and keep that traceability current.
• Run and own cluster validation end to end: cluster-wide performance, interconnect, storage, stability and burn‑in.
• Track validation status for every cluster with a clear view of pass/fail and blockers.
• Coordinate test windows with fabric, provisioning and OEM teams to avoid idle time.
• Read raw test output, question nonsensical results, and write or modify scripts when needed.
• Triage failures to the right owner with evidence and drive them to closure.
• Re‑run and verify fixes before marking anything as passed.
• Produce acceptance reports that tie results to criteria and support sign-off.
• Brief the customer and leadership on validation status and risk to handoff date.
• Feed learnings back into validation playbooks, test suites and automation.

Skills

Test frameworks

Python scripting

Bash scripting

GPU cluster performance

Cluster networking & storage testing

Data-driven validation

Clear communication

Tools

Python

Bash

Job description

Nscale is seeking a Validation TPM to transform customer acceptance criteria into scalable test plans and to drive end-to-end validation of large GPU clusters. You will work between engineering, OEM partners and the customer, delivering results that determine handoff readiness.

Bring 7-10+ years in software validation for infrastructure, strong scripting in Python/Bash, and deep knowledge of GPU cluster performance. Expect travel up to 50% to data center sites as part of the role.