Senior AI Infrastructure Validation Architect

Mainz Brady Group; Portland (OR); On-site; USD 150,000 - 190,000; Full time

Job summary

Mainz Brady Group is seeking a Senior AI Infrastructure Validation Engineer to validate the readiness, reliability, and performance of large-scale AI infrastructure environments before they reach production.

This role intersects systems engineering, networking, GPU infrastructure, container platforms, and distributed AI workloads, designing validation strategies that uncover failures early and accelerate root-cause analysis.

Qualifications

• 7+ years of experience in infrastructure validation, systems validation, performance engineering, infrastructure QA, HPC, or AI infrastructure certification.
• Strong hands-on troubleshooting experience across: Linux systems, GPU infrastructure, Network fabrics, Containers, Distributed workloads.
• Demonstrated experience designing validation strategies, frameworks, and qualification approaches, rather than only executing predefined test cases.
• Understanding of AI infrastructure dependencies including: NCCL, RDMA, Storage throughput, Multi-node communication, Distributed training behavior, Cluster orchestration.
• Ability to distinguish infrastructure defects from application, framework, workload, and configuration issues.
• Strong scripting and automation skills using Python, Bash, or similar technologies.
• Experience automating test execution, evidence collection, analysis, and reporting.
• Strong root-cause analysis and systems troubleshooting skills.
• Excellent written communication skills with experience producing detailed defect reports, readiness assessments, and technical recommendations.

Responsibilities

• Design and execute comprehensive validation plans for AI infrastructure spanning Compute nodes, GPU communication, Network fabric health, Storage access, Container orchestration, Distributed workload readiness.
• Perform structured bring-up, soak, regression, qualification, and certification testing for new and modified AI cluster environments.
• Reproduce, troubleshoot, and isolate failures involving: Distributed training workloads, GPU/node instability, NCCL and communication libraries, Kubernetes and container platforms, Storage paths and throughput, Ethernet and InfiniBand transport behavior.
• Validate Ethernet and InfiniBand environments, including host readiness, RDMA connectivity, and end-to-end workload behavior.
• Correlate failures across system logs, telemetry, firmware state, infrastructure health, and workload symptoms to accelerate root-cause analysis.
• Determine whether failures originate from infrastructure, hardware, networking, software frameworks, workloads, or configuration.
• Partner with Linux, networking, deployment, platform, and infrastructure engineering teams to close validation gaps before production handoff.
• Define defect signatures, validation criteria, pass/fail thresholds, readiness assessments, and release recommendations.
• Build and improve automation for: Cluster certification, Health scoring, Regression testing, Evidence collection, Post-change validation, Readiness reporting.
• Improve repeatability and scalability of validation processes across large AI infrastructure environments.

Skills

Linux systems

GPU infrastructure

Network fabrics

Containers

Distributed workloads

Python

Bash

Automation

Root-cause analysis

Technical documentation

Tools

Kubernetes

Containers

NCCL

RDMA

InfiniBand

Ethernet

Python

Bash

Telemetry Platforms

Cluster Certification Frameworks

Job description

Mainz Brady Group is seeking a Senior AI Infrastructure Validation Engineer to validate the readiness, reliability, and performance of large-scale AI infrastructure environments before they reach production.

This role intersects systems engineering, networking, GPU infrastructure, container platforms, and distributed AI workloads, designing validation strategies that uncover failures early and accelerate root-cause analysis.