GPU Performance & Validation Engineer

GTN Technical Staffing; Dallas (TX); On-site; USD 110,000 - 170,000; Full time

Job summary

GTN Technical Staffing seeks a GPU Performance & Validation Engineer to ensure large-scale GPU infrastructure is production-ready for AI/ML, HPC, and research workloads. You will design validation frameworks, run benchmarks, and collaborate with engineering to drive performance improvements across distributed GPU environments.

The role emphasizes automation, observability, and integration into CI/CD pipelines, with a focus on profiling, bottleneck analysis, and capacity planning.

Qualifications

• Experience validating, profiling, and tuning large-scale GPU clusters and accelerator-based infrastructure.
• Strong knowledge of NVIDIA ClusterKit, Nsight, DCGM, NVIDIA validation tools, and MLPerf, including GPU and DPU performance assessment.
• Experience benchmarking network and storage performance in InfiniBand and RoCE environments using ClusterKit, iPerf, or comparable tools.
• Hands‑on Linux benchmarking experience using Phoronix Test Suite or equivalent tools.
• Strong programming and automation skills using Python and Go, with experience developing tools for Kubernetes environments.
• Experience working in Ubuntu Linux environments.
• Experience building repeatable validation frameworks, micro‑benchmarks, and automated testing workflows.
• Strong knowledge of OpenTelemetry, Prometheus, ELK, and Grafana, including defining observability practices for HPC infrastructure.
• Experience supporting geographically distributed HPC environments and using performance data to guide architectural decisions.
• Ability to troubleshoot across GPU compute, Linux, networking, storage, and workload layers.
• Demonstrated ability to lead technical initiatives, document findings, and influence decisions across engineering and research teams.
• Ability to evaluate emerging technologies and translate findings into practical infrastructure recommendations.

Responsibilities

• Design and implement validation frameworks to assess GPU node and cluster readiness, functionality, utilization, and performance.
• Establish repeatable testing methodologies for AI/ML and HPC workloads across large-scale GPU clusters.
• Execute industry-standard and workload-specific benchmarks across GPU compute, storage, and networking.
• Profile workloads and investigate bottlenecks across hardware, operating systems, interconnects, and supporting infrastructure.
• Establish performance baselines and acceptance criteria for new deployments and infrastructure changes.
• Partner with engineering teams to resolve issues and verify improvements through repeatable testing.
• Evaluate throughput, latency, and scalability across InfiniBand and RoCE environments.
• Benchmark storage performance and assess its impact on GPU workload efficiency.
• Perform Linux system benchmarking to identify configuration gaps, resource constraints, and performance inconsistencies.
• Assess GPU and DPU performance using NVIDIA tools and other relevant benchmarking utilities.
• Compare results across nodes, clusters, and locations to identify deviations and optimization opportunities.

Skills

Python

Go

Linux

Performance testing

Education

Bachelor's degree in CS/Engineering

Tools

NVIDIA ClusterKit

Nsight

DCGM

MLPerf

iPerf

Phoronix Test Suite

Ubuntu Linux

OpenTelemetry

Prometheus

ELK

Grafana

Kubernetes

Job description

Our client is seeking a GPU Performance & Validation Engineer to ensure large-scale GPU infrastructure is production-ready and delivers consistent performance for demanding AI, machine learning, HPC, and research workloads.

This role combines GPU performance engineering, infrastructure benchmarking, automated validation, and observability across compute, storage, and networking. The engineer will develop repeatable testing frameworks, establish performance baselines, and investigate bottlenecks across distributed GPU environments.

Working within the Architecture organization, this individual will partner with engineering, infrastructure, and research teams to translate performance findings into improvements and inform future infrastructure decisions.

Key Responsibilities

GPU Performance & Production Validation

• Design and implement validation frameworks to assess GPU node and cluster readiness, functionality, utilization, and performance.
• Establish repeatable testing methodologies for AI/ML and HPC workloads across large-scale GPU clusters.
• Execute industry-standard and workload-specific benchmarks across GPU compute, storage, and networking.
• Profile workloads and investigate bottlenecks across hardware, operating systems, interconnects, and supporting infrastructure.
• Establish performance baselines and acceptance criteria for new deployments and infrastructure changes.
• Partner with engineering teams to resolve issues and verify improvements through repeatable testing.
• Evaluate throughput, latency, and scalability across InfiniBand and RoCE environments.
• Benchmark storage performance and assess its impact on GPU workload efficiency.
• Perform Linux system benchmarking to identify configuration gaps, resource constraints, and performance inconsistencies.
• Assess GPU and DPU performance using NVIDIA tools and other relevant benchmarking utilities.
• Compare results across nodes, clusters, and locations to identify deviations and optimization opportunities.

Automation & Validation Frameworks

• Develop automated validation tools and micro-benchmarking frameworks using Python, Go, and Kubernetes.
• Build repeatable workflows for test execution, results collection, analysis, and reporting.
• Integrate benchmarking and validation into CI/CD pipelines and infrastructure lifecycle processes.
• Maintain testing documentation, benchmark configurations, and reproducible evaluation procedures.
• Expand continuous validation coverage as hardware, software, and workload requirements evolve.

Observability & Performance Analysis

• Develop monitoring and dashboards using Prometheus, Grafana, OpenTelemetry, and ELK.
• Define telemetry requirements for GPU utilization, cluster health, network and storage performance, and workload behavior.
• Correlate benchmark results with infrastructure telemetry to support root-cause analysis.
• Establish observability practices for ongoing performance assessment and troubleshooting.

Architecture & Technical Collaboration

• Translate validation results into recommendations for infrastructure design, configuration, tuning, and capacity planning.
• Document hardware evaluations, testing methodologies, performance findings, and remediation recommendations.
• Collaborate with engineering, infrastructure, and research teams to align testing with production workload requirements.
• Evaluate emerging hardware, benchmarking tools, and architectures to support infrastructure planning.
• Lead technical investigations and communicate findings to technical and business stakeholders.

Required Qualifications

• Experience validating, profiling, and tuning large-scale GPU clusters and accelerator-based infrastructure.
• Strong knowledge of NVIDIA ClusterKit, Nsight, DCGM, NVIDIA validation tools, and MLPerf, including GPU and DPU performance assessment.
• Experience benchmarking network and storage performance in InfiniBand and RoCE environments using ClusterKit, iPerf, or comparable tools.
• Hands‑on Linux benchmarking experience using the Phoronix Test Suite or equivalent tools.
• Strong programming and automation skills using Python and Go, with experience developing tools for Kubernetes environments.
• Experience working in Ubuntu Linux environments.
• Experience building repeatable validation frameworks, micro‑benchmarks, and automated testing workflows.
• Strong knowledge of OpenTelemetry, Prometheus, ELK, and Grafana, including defining observability practices for HPC infrastructure.
• Experience supporting geographically distributed HPC environments and using performance data to guide architectural decisions.
• Ability to troubleshoot across GPU compute, Linux, networking, storage, and workload layers.
• Demonstrated ability to lead technical initiatives, document findings, and influence decisions across engineering and research teams.
• Ability to evaluate emerging technologies and translate findings into practical infrastructure recommendations.

Education

Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent professional experience.