Senior Validation TPM, GPU Cluster Readiness & Automation

Nscale; San Francisco, Seattle (CA, WA); On-site; USD 180,000 - 240,000; Full time

Job summary

Nscale is seeking a Senior Validation Technical Program Manager, Logical Deployment, to validate large GPU clusters before handoff to customers. You will translate acceptance criteria into scalable test plans, run end-to-end validation across cluster components, and deliver traceable results that prove readiness.

This role sits between engineering, OEM partners and customers and requires hands-on testing and scripting.

Qualifications

• 7-10+ years in software validation, test engineering or QA for infrastructure.
• Hands-on experience building and running test frameworks, and reading and debugging their output.
• Strong scripting in Python and Bash, and comfort modifying test harnesses and data pipelines.
• Working knowledge of GPU cluster performance and validation, such as NCCL collectives, HPL, DCGM diagnostics and burn‑in.
• Familiarity with cluster networking and storage testing, including InfiniBand or RoCE bandwidth and latency, and storage throughput.
• Experience turning requirements or acceptance criteria into test plans with clear pass/fail evidence.
• Clear, concise communication with engineers and customers, in writing and in the room.
• Ability to travel up to 50% of the time, primarily to data center sites.

Responsibilities

• Translate acceptance criteria into tests.
• Agree the plan with the customer and engineering before testing starts, so there are no surprises at handoff.
• Map every acceptance criterion to the test result that proves it, and keep that traceability current.
• Drive acceptance validation end to end: cluster-wide performance, interconnect, storage, stability and burn-in.
• Track validation status for every cluster, with a clear view of what has passed, what has failed and what is blocked.
• Coordinate test windows with fabric, provisioning and OEM teams so validation runs without idle time.
• Read raw test output, not just summaries. Question results that don’t make sense and find the root cause.
• Write or modify scripts when a test, harness or data collection step is missing or broken.
• Triage failures to the right owner and drive them to closure with evidence.
• Re-run and verify fixes before marking anything as passed.
• Produce acceptance reports that tie results to criteria, explain exceptions plainly and support sign-off.
• Brief the customer and leadership on validation status, with an honest view of risk to the handoff date.
• Feed what you learn back into the validation playbook, test suites and automation.

Skills

Software validation

Test engineering

Python scripting

Bash scripting

GPU cluster validation

Data center knowledge

Communication

Travel up to 50%

Tools

Test frameworks

NCCL collectives

HPL

DCGM diagnostics

InfiniBand/RoCE

Job description

Nscale is seeking a Senior Validation Technical Program Manager, Logical Deployment, to validate large GPU clusters before handoff to customers. You will translate acceptance criteria into scalable test plans, run end-to-end validation across cluster components, and deliver traceable results that prove readiness.

This role sits between engineering, OEM partners and customers and requires hands-on testing and scripting.