Work Arrangement: Hybrid
Employment Type: Direct Hire
Compensation: Competitive Base Salary + Performance Bonus
Overview
Our client, a growing high-performance computing (HPC) and AI cloud infrastructure company, is seeking a HPC Infrastructure Engineer – GPU & Firmware to develop and manage automation across a large-scale fleet of CPU servers, GPU systems, and rack-scale AI infrastructure.
This hands-on engineering role combines infrastructure automation, Linux systems engineering, firmware lifecycle management, and bare-metal provisioning. The engineer will own the processes and automation required to qualify, deploy, monitor, and maintain firmware across diverse hardware platforms while ensuring production environments remain secure, reliable, and scalable.
The environment includes NVIDIA A100/H200 GPU servers, HPE GB200 and Dell GB300 NVL72 rack-scale systems, NVIDIA BlueField DPUs, and ConnectX network adapters.
Working closely with Data Center Operations, Networking, Software Engineering, and hardware vendors, this individual will develop production-grade automation that simplifies hardware onboarding, minimizes manual intervention, and enables safe, repeatable firmware deployments across large-scale HPC and AI environments.
Key Responsibilities
Infrastructure & Fleet Automation
- Design, develop, and maintain automation for managing large-scale fleets of CPU servers, GPU systems, and rack-scale AI infrastructure.
- Build production-grade automation using Python, Ansible, Jenkins, Git, and workflow orchestration tools.
- Develop vendor-neutral automation utilizing Redfish, IPMI, HPE iLO, Dell iDRAC, HPE OneView, and NVIDIA management utilities.
- Create reusable workflows and declarative configurations for hardware provisioning, firmware updates, validation, and lifecycle management.
- Implement rack-aware deployment sequencing, controlled concurrency, retry mechanisms, failure handling, and recovery procedures.
- Integrate hardware inventory systems, management APIs, and automated health checks into infrastructure workflows.
- Identify opportunities to eliminate manual processes and improve fleet reliability, consistency, and operational efficiency.
Firmware Lifecycle Management & Qualification
- Own firmware lifecycle management from vendor release evaluation through testing, approval, production deployment, and ongoing maintenance.
- Evaluate firmware updates, security advisories, compatibility requirements, and potential operational risks.
- Perform firmware qualification and validation in representative lab environments prior to production deployment.
- Maintain approved firmware baselines, version-controlled release manifests, and auditable deployment records.
- Investigate firmware regressions, compatibility issues, and hardware failures in collaboration with vendors and engineering teams.
- Develop automated processes to identify and remediate firmware drift across servers, racks, and data centers.
Firmware Packaging & Production Deployment
- Build, maintain, and deploy firmware packages, including custom HPE Service Packs for ProLiant (SPP) and NVIDIA BlueField BFB bundles.
- Develop automated firmware deployment pipelines with controlled artifact promotion and release approval processes.
- Execute safe production rollouts using lab qualification, canary deployments, phased upgrades, and automated health validation.
- Implement deployment safeguards, rollback or recovery procedures, and failure remediation workflows.
- Ensure firmware deployment processes are scalable, repeatable, and minimally disruptive to production workloads.
Bare-Metal Provisioning & Hardware Onboarding
- Automate hardware onboarding and provisioning for new CPU/GPU servers and rack-scale infrastructure.
- Develop workflows for hardware discovery, inventory validation, management interface configuration, and firmware readiness.
- Support provisioning environments utilizing OpenStack/Ironic, Terraform, and cloud-init, where applicable.
- Collaborate with data center engineering and operations teams to ensure new infrastructure is production-ready.
- Improve hardware deployment consistency through standardized configurations, automated validation, and infrastructure-as-code practices.
Infrastructure Reliability, Security & Collaboration
- Incorporate secrets management, least-privilege access controls, hardware-health telemetry, and actionable monitoring into automation workflows.
- Troubleshoot complex hardware, Linux, firmware, and infrastructure automation issues.
- Partner with hardware manufacturers and vendors to investigate defects, coordinate escalations, and validate technical resolutions.
- Collaborate with Infrastructure, Networking, Software Engineering, and Data Center Operations teams on platform reliability and deployment initiatives.
- Establish engineering standards, operating procedures, documentation, and troubleshooting guides.
- Drive continuous improvements in infrastructure automation, deployment safety, and fleet operational efficiency.
Required Qualifications
- Strong experience in infrastructure automation, systems engineering, or hardware fleet management within HPC, AI infrastructure, cloud computing, or large-scale data center environments.
- Hands-on experience managing heterogeneous server environments, including CPU/GPU systems and enterprise server hardware.
- Experience with firmware lifecycle management, including qualification, compatibility testing, packaging, deployment, and version control.
- Strong Python programming skills with demonstrated experience developing production automation.
- Strong Linux systems administration and troubleshooting experience.
- Experience with Ansible, Jenkins, Git, and automated deployment pipelines.
- Hands-on experience with Redfish, IPMI, and hardware management interfaces such as HPE iLO, Dell iDRAC, or HPE OneView.
- Experience automating firmware upgrades, server configuration, hardware provisioning, or infrastructure lifecycle workflows.
- Experience with workflow orchestration tools such as Temporal, including concurrency management, retry logic, failure handling, and recovery.
- Familiarity with firmware packaging, release management, and controlled production deployment practices.
- Experience implementing canary deployments, phased rollouts, automated validation, and recovery procedures.
- Demonstrated ability to troubleshoot hardware and firmware issues and collaborate directly with equipment vendors.
- Strong technical ownership, problem-solving, documentation, and cross-functional communication skills.
Preferred Qualifications
- Experience supporting NVIDIA GPU infrastructure, including A100, H200, GB200, or GB300 systems.
- Familiarity with NVIDIA BlueField DPUs, ConnectX adapters, and associated firmware management utilities.
- Experience managing firmware across HPE, Dell, NVIDIA, or other enterprise hardware platforms.
- Experience with OpenStack/Ironic for bare-metal provisioning and lifecycle management.
- Familiarity with Terraform, cloud-init, and infrastructure-as-code methodologies.
- Experience developing hardware inventory integrations, automated firmware drift detection, and hardware-health monitoring.
- Background supporting large-scale HPC clusters, GPU-as-a-Service (GPUaaS), AI training infrastructure, or cloud computing platforms.
- Experience building reliable automation services with comprehensive logging, monitoring, and failure recovery capabilities.
Education
Bachelor’s degree in Computer Engineering, Electrical Engineering, Computer Science, Information Technology, or a related technical discipline, or equivalent practical experience.