Senior Validation Technical Program Manager, Logical Deployment

nscaleoperationsukltd; San Francisco (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

Nscale is seeking a Validation TPM to transform customer acceptance criteria into scalable test plans and to drive end-to-end validation of large GPU clusters. You will work between engineering, OEM partners and the customer, delivering results that determine handoff readiness.

Bring 7-10+ years in software validation for infrastructure, strong scripting in Python/Bash, and deep knowledge of GPU cluster performance. Expect travel up to 50% to data center sites as part of the role.

Qualifications

• 7-10+ years in software validation, test engineering or QA for infrastructure or distributed systems, including program or project ownership.
• Hands-on experience building and running test frameworks, and reading and debugging their output.
• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn‑in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, in writing and in the room.
• Ability to travel up to 50% of the time, primarily to data center sites.

Responsibilities

• Turn acceptance criteria into tests: translate each customer's acceptance criteria into a clear test plan, what is run, at what scale, and with what pass thresholds.
• Agree the plan with the customer and engineering before testing starts.
• Map every acceptance criterion to the test result that proves it, and keep that traceability current.
• Run and own cluster validation end to end: cluster-wide performance, interconnect, storage, stability and burn‑in.
• Track validation status for every cluster with a clear view of pass/fail and blockers.
• Coordinate test windows with fabric, provisioning and OEM teams to avoid idle time.
• Read raw test output, question nonsensical results, and write or modify scripts when needed.
• Triage failures to the right owner with evidence and drive them to closure.
• Re‑run and verify fixes before marking anything as passed.
• Produce acceptance reports that tie results to criteria and support sign-off.
• Brief the customer and leadership on validation status and risk to handoff date.
• Feed learnings back into validation playbooks, test suites and automation.

Skills

Test frameworks

Python scripting

Bash scripting

GPU cluster performance

Cluster networking & storage testing

Data-driven validation

Clear communication

Tools

Python

Bash

Job description

About Nscale and the Deployment PMO

Nscale builds and operates large-scale AI infrastructure, delivering GPU clusters for leading hyperscalers, AI labs and enterprises across North America, EMEA and APAC.

The Deployment PMO turns signed contracts into live AI infrastructure, coordinating the many teams and partners involved in bringing large GPU clusters online.

About the role

The Validation TPM owns the proof that a cluster is ready for the customer. You take the customer's acceptance criteria and turn them into a test plan, run it at cluster scale, and deliver results that stand up to scrutiny. You track the state of validation for every cluster, from first test run to signed acceptance.

You come from software validation and test frameworks, not only program management. You go deep into any test, question outputs that look too good or too bad, and write the script yourself when that is the fastest way to unblock testing. You sit between engineering, OEM partners and the customer, and your results decide when a cluster is handed off.

This is an individual contributor role reporting to the Director, Logical Deployment. Expect to travel up to 50% of the time, primarily to data center sites.

What you'll do

Turn acceptance criteria into tests

Translate each customer's acceptance criteria into a clear test plan: what is run, at what scale, and with what pass thresholds.

Agree the plan with the customer and engineering before testing starts, so there are no surprises at handoff.

Map every acceptance criterion to the test result that proves it, and keep that traceability current.

Run and own cluster validation

Drive acceptance validation end to end: cluster-wide performance, interconnect, storage, stability and burn-in.

Track validation status for every cluster, with a clear view of what has passed, what has failed and what is blocked.

Coordinate test windows with fabric, provisioning and OEM teams so validation runs without idle time.

Go deep and unblock

Read raw test output, not just summaries. Question results that don’t make sense and find the root cause.

Write or modify scripts when a test, harness or data collection step is missing or broken.

Triage failures to the right owner (hardware, fabric, firmware, software or configuration) with evidence, and drive them to closure.

Re‑run and verify fixes before marking anything as passed.

Close out with the customer

Produce acceptance reports that tie results to criteria, explain exceptions plainly and support sign‑off.

Brief the customer and leadership on validation status, with an honest view of risk to the handoff date.

Feed what you learn back into the validation playbook, test suites and automation, so each cluster validates faster than the last.

What success looks like in your first 6-12 months

• Every active cluster has a test plan mapped to its customer's acceptance criteria, agreed before testing starts.
• Validation status is visible for every cluster without manual chasing.
• Failures are triaged to an owner with evidence within a day, and no cluster reaches handoff with an unexplained result.
• Customers sign off on acceptance reports without rework.
• A reusable acceptance test suite and reporting automation are in use on every new cluster.

What you bring

• 7-10+ years in software validation, test engineering or QA for infrastructure or distributed systems, including program or project ownership.
• Hands‑on experience building and running test frameworks, and reading and debugging their output.
• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn‑in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, in writing and in the room.
• Ability to travel up to 50% of the time, primarily to data center sites.

How you work

Rigorous: You trust data, not dashboards. A pass needs evidence, and a strange number gets investigated, not explained away.

Bold: You call a cluster not ready when the results say so, even close to a handoff date, and you put a path to ready next to it.

Assertive with partners: You hold engineering, OEM partners and vendors to fixing what the tests find. You **escalate early**, directly and with evidence.

Accountable: You own validation outcomes. Rather than "waiting on engineering", you explain what you are doing to unblock it.

Steady under pressure: You keep testing moving when failures stack up and the handoff date is close.

Technically deep: You can read a test log, a performance trace or a script and know where the problem is.

Preferred qualifications

Experience validating NVIDIA GB200/GB300 or Vera Rubin NVL72-cla