Senior Validation Technical Program Manager, Logical Deployment

Nscale; San Francisco, Seattle (CA, WA); On-site; USD 180,000 - 240,000; Full time

Job summary

Nscale is seeking a Senior Validation Technical Program Manager, Logical Deployment, to validate large GPU clusters before handoff to customers. You will translate acceptance criteria into scalable test plans, run end-to-end validation across cluster components, and deliver traceable results that prove readiness.

This role sits between engineering, OEM partners and customers and requires hands-on testing and scripting.

Qualifications

• 7-10+ years in software validation, test engineering or QA for infrastructure.
• Hands-on experience building and running test frameworks, and reading and debugging their output.
• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn‑in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, in writing and in the room.
• Ability to travel up to 50% of the time, primarily to data center sites.

Responsibilities

• Translate acceptance criteria into tests.
• Agree the plan with the customer and engineering before testing starts, so there are no surprises at handoff.
• Map every acceptance criterion to the test result that proves it, and keep that traceability current.
• Drive acceptance validation end to end: cluster-wide performance, interconnect, storage, stability and burn-in.
• Track validation status for every cluster, with a clear view of what has passed, what has failed and what is blocked.
• Coordinate test windows with fabric, provisioning and OEM teams so validation runs without idle time.
• Read raw test output, not just summaries. Question results that don’t make sense and find the root cause.
• Write or modify scripts when a test, harness or data collection step is missing or broken.
• Triage failures to the right owner and drive them to closure with evidence.
• Re-run and verify fixes before marking anything as passed.
• Produce acceptance reports that tie results to criteria, explain exceptions plainly and support sign-off.
• Brief the customer and leadership on validation status, with an honest view of risk to the handoff date.
• Feed what you learn back into the validation playbook, test suites and automation.

Skills

Software validation

Test engineering

Python scripting

Bash scripting

GPU cluster validation

Data center knowledge

Communication

Travel up to 50%

Tools

Test frameworks

NCCL collectives

HPL

DCGM diagnostics

InfiniBand/RoCE

Job description

Senior Validation Technical Program Manager, Logical Deployment

About Nscale and the Deployment PMO

Nscale builds and operates large-scale AI infrastructure, delivering GPU clusters for leading hyperscalers, AI labs and enterprises across North America, EMEA and APAC.

The Deployment PMO turns signed contracts into live AI infrastructure, coordinating the many teams and partners involved in bringing large GPU clusters online.

About the role

The Validation TPM owns the proof that a cluster is ready for the customer. You take the customer’s acceptance criteria and turn them into a test plan, run it at cluster scale, and deliver results that stand up to scrutiny. You track the state of validation for every cluster, from first test run to signed acceptance.

You come from software validation and test frameworks, not only program management. You go deep into any test, question outputs that look too good or too bad, and write the script yourself when that is the fastest way to unblock testing. You sit between engineering, OEM partners and the customer, and your results decide when a cluster is handed off.

This is an individual contributor role reporting to the Director, Logical Deployment. Expect to travel up to 50% of the time, primarily to data center sites.

What you’ll do

Turn acceptance criteria into tests

• Translate each customer’s acceptance criteria into a clear test plan: what is run, at what scale, and with what pass thresholds.
• Agree the plan with the customer and engineering before testing starts, so there are no surprises at handoff.
• Map every acceptance criterion to the test result that proves it, and keep that traceability current.

Run and own cluster validation

• Drive acceptance validation end to end: cluster-wide performance, interconnect, storage, stability and burn-in.
• Track validation status for every cluster, with a clear view of what has passed, what has failed and what is blocked.
• Coordinate test windows with fabric, provisioning and OEM teams so validation runs without idle time.

Go deep and unblock

• Read raw test output, not just summaries. Question results that don’t make sense and find the root cause.
• Write or modify scripts when a test, harness or data collection step is missing or broken.
• Triage failures to the right owner (hardware, fabric, firmware, software or configuration) with evidence, and drive them to closure.
• Re-run and verify fixes before marking anything as passed.

Close out with the customer

• Produce acceptance reports that tie results to criteria, explain exceptions plainly and support sign-off.
• Brief the customer and leadership on validation status, with an honest view of risk to the handoff date.
• Feed what you learn back into the validation playbook, test suites and automation, so each cluster validates faster than the last.

What success looks like in your first 6–12 months

• Every active cluster has a test plan mapped to its customer’s acceptance criteria, agreed before testing starts.
• Validation status is visible for every cluster without manual chasing.
• Failures are triaged to an owner with evidence within a day, and no cluster reaches handoff with an unexplained result.
• Customers sign off on acceptance reports without rework.
• A reusable acceptance test suite and reporting automation are in use on every new cluster.

What you bring

• 7-10+ years in software validation, test engineering or QA for infrastructure or distributed systems, including program or project ownership.
• Hands‑on experience building and running test frameworks, and reading and debugging their output.
• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn‑in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, in writing and in the room.
• Ability to travel up to 50% of the time, primarily to data center sites.

How you work

• Rigorous: You trust data, not dashboards. A pass needs evidence, and a strange number gets investigated, not explained away.
• Bold: You call a cluster not ready when the results say so, even close to a handoff date, and you put a path to ready next to it.
• Assertive with partners: You hold engineering, OEM partners and vendors to fixing what the tests find. You escape early, directly and with evidence.
• Accountable: You own validation outcomes. Rather than “waiting on engineering,” you explain what you are doing to unblock it.
• Steady under pressure: You keep testing moving when failures stack up and the handoff date is close.
• Technically deep: You can read a test log, a performance trace or a script and know where the problem is.

Preferred qualifications

• Experience validating NVIDIA GB200/GB300 or Vera Rubin NVL72-class clusters, including NVIDIA cluster validation tooling.
• Experience with MLPerf or customer-defined training and inference benchmarks at scale.
• Familiarity with Slurm or Kubernetes for scheduling large test jobs.
• Experience at a hyperscaler, neocloud or AI lab accepting 10K+ GPU clusters.
• Background in CI/CD and test automation for infrastructure.

For information on how Nscale handles candidate personal data, please see our Employee & Candidate Privacy Notice: Here.

Nscale does not accept unsolicited candidate submissions from recruitment agencies.