Principal Software Engineer – Rack-Scale System Software

Jobtailor; California (MO); On-site; USD 180,000 - 260,000; Full time

Job summary

Jobtailor is seeking a senior technologist to drive rack-scale system software and firmware architecture across CSP engagements in the United States. The role focuses on fabric management, NVSwitch-like behaviors, health telemetry, and recovery workflows across fleets.

You will mentor customer engineering teams, influence architecture decisions, and collaborate with multi-functional teams to ensure customer requirements are reflected in software and firmware development.

Qualifications

• 15+ years in system software, firmware, or large-scale distributed systems.
• BS/MS in CS/EE or equivalent.
• Deep knowledge of rack-scale software challenges, multi-component coordination, and serviceability.
• Experience with fabric management, cluster management, or system orchestration.
• Familiar with firmware architectures and update lifecycle management.
• Understanding of error handling, fault isolation, and graceful degradation.
• Experience with health telemetry and fleet observability.
• Knowledge of GPU/accelerator system software is a plus.
• Customer-focused with ability to mentor across teams.
• Strong communication translating architecture to customer teams.

Responsibilities

• Align rack-scale SW/FW architecture across CSP engagements, including fabric management, health monitoring, and firmware-driven recovery.
• Lead technical work streams with CSP engineering teams on rack-scale system software.
• Capture CSP feedback on APIs, workflows, and recovery behavior and influence architecture decisions.
• Collaborate with cross-functional teams to reflect customer requirements in software/features.
• Identify cross-CSP patterns and drive documentation, tooling, and test strategy improvements.
• Collaborate with execution teams on early left-shift strategy before hardware availability.
• Make critical technical decisions on SW/FW tradeoffs and mitigate risks via early engagement.

Skills

System software

Firmware

Distributed systems

Leadership

Communication

Observability

Education

BS in CS/EE or related

MS in CS/EE

Tools

Cluster management

Telemetry systems

Firmware update lifecycle

Job description

Responsibilities

• Drive rack-scale SW/FW architecture alignment across CSP engagements — including fabric management software, link health monitoring, GPU/NVSwitch error handling, SW/FW serviceability features (e.g., hot-plug support, component isolation, firmware-driven recovery), and multi-component firmware orchestration.
• Drive technical work streams with CSP engineering teams on rack-scale system software — ensuring they deeply understand fabric management, NVSwitch behavior, error handling and recovery policies, health telemetry APIs, and SW/FW-controlled recovery operation.
• Capture and synthesize CSP engineering feedback on rack-scale system software — health monitoring APIs, SW-driven serviceability workflows, firmware update orchestration, and error recovery behavior — champion that feedback into NVIDIA's architecture decisions.
• Collaborate with multi-functional teams to ensure customer operational requirements are reflected in system software and firmware development.
• Identify cross-CSP patterns in rack-scale SW/FW issues, error handling behavior, and system configuration practices — drive documentation, tooling, and test strategy improvements as a result.
• Collaborate with execution teams on left-shift strategy — ensuring customer-side SW/FW integration work is identified early and completed ahead of hardware availability.
• Make critical technical decisions on rack-scale system SW/FW tradeoffs and mitigate execution risks through early engagement with CSP engineering teams.

Requirements

• 15+ years of experience in system software, platform firmware, or large-scale distributed systems engineering.
• BS or MS in Computer Science, Electrical Engineering, or related field (or equivalent experience).
• Deep understanding of rack-scale system software challenges: multi-component coordination, error propagation, health monitoring, and serviceability / reliability.
• Experience with fabric management software, cluster management, or system-level orchestration frameworks.
• Familiarity with firmware architectures and update lifecycle management (multi-component update sequencing, rollback, recovery).
• Understanding of error handling and recovery design patterns in distributed systems — fault isolation, retry policies, graceful degradation.
• Experience with health monitoring and telemetry systems: health scoring, event correlation, API design for fleet-level observability.
• Understanding of GPU or accelerator system software (drivers, device management, power management) is a strong plus.
• Customer obsession — genuine passion for understanding how CSPs operate sophisticated systems at fleet scale and simplifying their experience.
• Proven success providing technical leadership across organizational boundaries and influencing system software design without direct authority.
• Strong communication — ability to translate complex system software architecture into actionable mentorship for customer engineering teams.