GPU Performance & Validation Engineer - Large-Scale AI/ML

GTN Technical Staffing; Dallas (TX); On-site; USD 110,000 - 170,000; Full time

Job summary

GTN Technical Staffing seeks a GPU Performance & Validation Engineer to ensure large-scale GPU infrastructure is production-ready for AI/ML, HPC, and research workloads. You will design validation frameworks, run benchmarks, and collaborate with engineering to drive performance improvements across distributed GPU environments.

The role emphasizes automation, observability, and integration into CI/CD pipelines, with a focus on profiling, bottleneck analysis, and capacity planning.

Qualifications

• Experience validating, profiling, and tuning large-scale GPU clusters and accelerator-based infrastructure.
• Strong knowledge of NVIDIA ClusterKit, Nsight, DCGM, NVIDIA validation tools, and MLPerf, including GPU and DPU performance assessment.
• Experience benchmarking network and storage performance in InfiniBand and RoCE environments using ClusterKit, iPerf, or comparable tools.
• Hands‑on Linux benchmarking experience using Phoronix Test Suite or equivalent tools.
• Strong programming and automation skills using Python and Go, with experience developing tools for Kubernetes environments.
• Experience working in Ubuntu Linux environments.
• Experience building repeatable validation frameworks, micro‑benchmarks, and automated testing workflows.
• Strong knowledge of OpenTelemetry, Prometheus, ELK, and Grafana, including defining observability practices for HPC infrastructure.
• Experience supporting geographically distributed HPC environments and using performance data to guide architectural decisions.
• Ability to troubleshoot across GPU compute, Linux, networking, storage, and workload layers.
• Demonstrated ability to lead technical initiatives, document findings, and influence decisions across engineering and research teams.
• Ability to evaluate emerging technologies and translate findings into practical infrastructure recommendations.

Responsibilities

• Design and implement validation frameworks to assess GPU node and cluster readiness, functionality, utilization, and performance.
• Establish repeatable testing methodologies for AI/ML and HPC workloads across large-scale GPU clusters.
• Execute industry-standard and workload-specific benchmarks across GPU compute, storage, and networking.
• Profile workloads and investigate bottlenecks across hardware, operating systems, interconnects, and supporting infrastructure.
• Establish performance baselines and acceptance criteria for new deployments and infrastructure changes.
• Partner with engineering teams to resolve issues and verify improvements through repeatable testing.
• Evaluate throughput, latency, and scalability across InfiniBand and RoCE environments.
• Benchmark storage performance and assess its impact on GPU workload efficiency.
• Perform Linux system benchmarking to identify configuration gaps, resource constraints, and performance inconsistencies.
• Assess GPU and DPU performance using NVIDIA tools and other relevant benchmarking utilities.
• Compare results across nodes, clusters, and locations to identify deviations and optimization opportunities.

Skills

Python

Go

Linux

Performance testing

Education

Bachelor's degree in CS/Engineering

Tools

NVIDIA ClusterKit

Nsight

DCGM

MLPerf

iPerf

Phoronix Test Suite

Ubuntu Linux

OpenTelemetry

Prometheus

ELK

Grafana

Kubernetes

Job description

GTN Technical Staffing seeks a GPU Performance & Validation Engineer to ensure large-scale GPU infrastructure is production-ready for AI/ML, HPC, and research workloads. You will design validation frameworks, run benchmarks, and collaborate with engineering to drive performance improvements across distributed GPU environments.

The role emphasizes automation, observability, and integration into CI/CD pipelines, with a focus on profiling, bottleneck analysis, and capacity planning.