Senior Hardware Diagnostics Engineer: Fleet Firmware

TensorWave Inc.; Las Vegas (NV); On-site; USD 150,000 - 230,000; Full time

Benefits offered by this job

Stock options

Medical, dental, and vision insurance

HSA contributions

Disability insurance

Life insurance

Pet & Legal insurance

Flexible PTO

Paid holidays

Parental leave

401(k)

Employee assistance program

Flexible spending account

In-office perks

Job summary

TensorWave Inc. seeks a Senior Hardware Diagnostics Engineer to own firmware validation and fleet-wide rollout across BIOS, BMC, NIC, GPU, and storage controllers.

You will define passing burn-in criteria, build validation harnesses, and drive RCAs with vendors to resolution. You will lead root cause analysis of hardware and firmware failures, work with OEM/ODM silicon vendors, and prepare next-gen AMD platform bring-up while mentoring the diagnostics team and documenting runbooks and best

Qualifications

• 6+ years in infrastructure engineering, hardware validation or related field.
• Hands-on with BMCs and out-of-band management (Redfish/IPMI).
• Firmware flashing, qualifying, and troubleshooting BIOS/BMC/device firmware.
• Strong Linux skills: boot, drivers, and hardware diagnostics tools.
• Ability to read PCIe signals, sensors, and telemetry data.
• Python automation against REST APIs and vendor tooling.
• Rigorous diagnostic method with evidence-based conclusions.
• Experience driving hardware issues with vendors to resolution.
• Clear technical writing for runbooks and RCAs.

Responsibilities

• Own firmware lifecycle across BIOS, BMC, NIC, GPU, and storage controllers.
• Design and build a validation harness for firmware releases.
• Build automation against Redfish, IPMI, and vendor APIs for fleet-scale work.
• Define pass criteria for burn-in tests and evidence retention.
• Ensure safe, staged rollout with rollback paths and blast radius control.
• Lead root cause analysis on hardware/firmware failures and telemetry.
• Escalate with OEM/ODM vendors and track fixes to closure.
• Prepare validation for next-gen AMD platforms before hardware lands.
• Mentor the diagnostics engineer and develop runbooks.
• Participate in incident response and RCAs.

Skills

Infrastructure engineering

Hardware validation

Linux systems

Firmware engineering

Python automation

Diagnostics method

Tools

Redfish

IPMI

iDRAC/iLO

Vendor tooling

Job description

About TensorWave

Our mission is simple: deliver seamless, secure, reliable, and resilient AI compute at scale. We've built a versatile cloud platform that eliminates infrastructure barriers, empowering builders to focus on innovation instead of fighting their stack. Because breakthrough AI should move at the speed of ideas, not infrastructure.

About the Role

We are building and operating large-scale infrastructure platforms to support high-performance AI workloads across multiple data centers. Our environment includes GPU-intensive systems, high-throughput networking, and rapidly scaling compute clusters.

We are looking for a Senior Hardware Diagnostics Engineer to own that surface: which versions we run, how we prove a version is safe, and how we push it across a fleet without breaking production. You'll set the bar for what validated hardware means here, build the tooling that enforces it, and be the person we escalate to when a failure doesn't make sense.

Firmware is where a lot of our hardest problems live. A BIOS revision changes how a GPU enumerates. A BMC update fixes one thing and breaks telemetry.

What You’ll Do

• Own the firmware lifecycle across BIOS, BMC, NIC, GPU, and storage controllers: version qualification, approved baselines, staged rollout, and rollback

• Design and build the validation harness that qualifies a firmware release before it reaches production, including regression and performance comparison between versions

• Build automation against Redfish, IPMI, and vendor management APIs so firmware and validation work runs at fleet scale rather than node by node

• Define what passing burn-in means: the test matrix, the thresholds, and the evidence we keep

• Make fleet-wide changes safe and predictable through pre-flight validation, blast radius control, staged waves, and a rollback path that has actually been tested

• Lead root cause analysis on hardware and firmware failures nobody else can reproduce, including PCIe error analysis, sensor and event log forensics, and thermal and power telemetry

• Own technical escalation with OEM, ODM, and silicon vendors on firmware defects, and track fixes to closure

• Prepare validation and bring-up capability for next-generation AMD platforms before the hardware lands

• Mentor the diagnostics engineer on the team, and turn recurring findings into runbooks and tooling

• Participate in incident response and root cause analysis

• Contribute to runbooks, documentation, and operational best practices

Who You Are

Required Qualifications

• 6+ years in infrastructure engineering, hardware validation, systems engineering, or a closely related field

• Deep hands-on experience with BMCs and out-of-band management — Redfish, IPMI, iDRAC, iLO, or equivalent — including scripted interaction, not clicking through a web UI

• Direct firmware experience flashing, qualifying, and troubleshooting BIOS, BMC, and device firmware on enterprise servers, and a clear sense of how badly this can go wrong at scale

• Strong Linux systems skills: boot process, kernel and driver behavior, and hardware diagnostics such as {{dmesg}}, {{lspci}}, {{ipmitool}}, SMART, and vendor CLIs

• Ability to read low-level hardware signals: PCIe topology and error output, SEL and sensor data, thermal and power telemetry

• Python for automation against REST APIs — Redfish, vendor tooling, and internal systems

• A rigorous diagnostic method: you form a hypothesis, design a test that can disprove it, isolate one variable at a time, and document the evidence behind your conclusion

• Experience driving hardware issues with vendors through to resolution

• Clear technical writing for runbooks, RCAs, and vendor escalations

Preferred Qualifications

• AMD GPU and ROCm experience: {{amd-smi}}, {{rocm-smi}}, ROCm Validation Suite

• GPU and HPC validation at scale: stress testing, collective communication testing with RCCL or NCCL, node health frameworks

• Low-level bus and firmware work: I2C, SMBus, BMC firmware behavior, Redfish schema extensions

• Kernel-level debugging, including crash dump analysis

• Ansible for fleet operations, and Git-based workflows with CI/CD

• NetBox or other DCIM and IPAM tooling as a source of truth

• Prior work at an OEM, ODM, or hyperscaler on hardware bring-up or validation

First Six Months

• By 90 days you'll have qualified and rolled out a firmware release using our existing process — and told us what's wrong with it. By six months, a validation step that used to be manual runs automatically, and we can state our approved firmware baseline across the fleet without anyone checking node by node.

What We Offer

• Stock Options

• 100% paid Medical, Dental, and Vision insurance for Employees

• Company Health Savings Account Contributions

• 100% paid Short Term and Long Term Disability Insurance for Employees

• Life and Voluntary Supplemental Insurance Options

• Other Insurance Options, such as Pet & Legal Insurance

• Various Supplementary Health Benefits, such as discounted Virtual Healthcare Appointments and Serious Illness Support

• Flexible Spending Account

• 401(k)

• Employee Assistance Program

• Flexible PTO

• Paid Holidays

• Parental Leave

• Other In-Office Perks

Equal Employment Opportunity

TensorWave is an Equal Opportunity Employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of any protected status under applicable law.

Reasonable Accommodations

TensorWave provides reasonable accommodations in accordance with applicable laws. If you require accommodation during the hiring process, please contact [e-mail].

Employment Eligibility

All offers of employment are contingent upon verification of identity and authorization to work in United States, as required by law.

Background Checks

Where permitted by law, employment may be contingent upon the successful completion of a job-related background check.

Data Privacy Notice

By submitting an application, you acknowledge that TensorWave may collect, use, and retain your personal information for recruiting and employment-related purposes in accordance with applicable data privacy laws.