SRE Triage Engineer

Quadrant Technologies; Redmond; On-site; AUD 120,000 - 180,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

Paid time off (PTO)

Professional development and training

Job summary

Quadrant Technologies is seeking a hands-on Linux Systems / Bare-Metal Infrastructure / SRE Triage Engineer to support 24/7 Follow-the-Sun infrastructure operations for large-scale AI/TPU superpod environments. The ideal candidate will have strong Linux administration, bare-metal provisioning, datacenter networking, Kubernetes, and infrastructure troubleshooting experience.

The primary responsibility is to rapidly identify and isolate hardware and software failures during automated machine

Qualifications

• Strong hands-on experience with Linux systems, particularly Ubuntu and Debian.
• Experience with PXE/iPXE, OS installation, Subiquity/autoinstall, UEFI, and BIOS.
• Knowledge of systemd, dmesg, journalctl, and Linux troubleshooting.
• Experience with BMC, Redfish, IPMI, and serial console management.

Responsibilities

• Troubleshoot Linux-based bare-metal server provisioning and OS installation issues.
• Diagnose infrastructure failures involving DHCP, DNS, mTLS certificates, BMC, PXE boot, and automated deployment workflows.
• Perform system-level troubleshooting using SSH, kubectl, journalctl, dmesg, Python, and Bash.
• Support datacenter networking including IPv4/IPv6, DHCP, DNS, NTP, and network connectivity.
• Monitor Kubernetes workloads and infrastructure health using kubectl, Temporal, Grafana, and ClickHouse.
• Analyze hardware qualification and burn-in test results to identify hardware-related failures.
• Differentiate software/infrastructure issues from physical hardware defects.
• Generate hardware repair tickets with rack/RU/machine/port details and required actions.
• Execute documented troubleshooting procedures, runbooks, and escalation processes.
• Collaborate with onsite datacenter technicians and global infrastructure teams.

Skills

Linux

Kubernetes

Bash

Python

SSH

Tools

PXE/iPXE

Subiquity/autoinstall

UEFI/BIOS

BMC/Redfish/IPMI

Serial console

Temporal

Grafana

ClickHouse

NetBox/Foundry

Job description

Quadrant Technologies is seeking a hands-on Linux Systems / Bare-Metal Infrastructure / SRE Triage Engineer to support 24/7 Follow-the-Sun infrastructure operations for large-scale AI/TPU superpod environments.

The ideal candidate will have strong Linux administration, bare-metal provisioning, datacenter networking, Kubernetes, and infrastructure troubleshooting experience.

The primary responsibility is to rapidly identify and isolate hardware and software failures during automated machine turnup workflows, execute documented runbook remediations, and coordinate actionable hardware repair requests with onsite datacenter teams.

Key Responsibilities

• Troubleshoot Linux-based bare-metal server provisioning, network boot, and OS installation issues.
• Diagnose infrastructure failures involving DHCP, DNS, mTLS certificates, BMC, PXE boot, and automated deployment workflows.
• Perform system-level troubleshooting using SSH, kubectl, journalctl, dmesg, Python, and Bash.
• Support datacenter networking, including IPv4/IPv6, DHCP, DNS, NTP, optical links, and network connectivity.
• Monitor Kubernetes workloads, distributed workflows, and infrastructure health using kubectl, Temporal, Grafana, and ClickHouse.
• Analyze hardware qualification and burn-in test results to identify optical, DIMM, PCIe, power, and thermal-related failures.
• Differentiate software/infrastructure issues from physical hardware defects.
• Generate detailed hardware repair tickets identifying rack, rack unit (RU), machine, port, and required corrective action.
• Execute documented troubleshooting procedures, runbooks, and escalation processes.
• Collaborate with onsite datacenter technicians and global infrastructure operations teams.

Required Qualifications

• Strong hands-on experience with Linux systems, particularly Ubuntu and Debian.
• Experience with PXE/iPXE, OS installation, Subiquity/autoinstall, UEFI, and BIOS.
• Knowledge of systemd, dmesg, journalctl, and Linux troubleshooting.
• Experience with BMC, Redfish, IPMI, and serial console management.

Datacenter Networking & Infrastructure

• Understanding of IPv4/IPv6, DHCP/DHCPv6, DNS, NTP, and network connectivity.
• Knowledge of PKI, mTLS certificates, and network authentication.
• Familiarity with ToR switches, optical circuit switches, fiber connectivity, and transceivers.

Kubernetes & Workflow Operations

• Hands-on experience with kubectl for pod inspection, log analysis, and troubleshooting.
• Familiarity with Teleport bastion access and remote infrastructure management.
• Experience with workflow orchestration and observability tools such as Temporal, Grafana, and ClickHouse.

Hardware & Software Troubleshooting

• Ability to identify hardware and software failures across bare-metal infrastructure.
• Experience troubleshooting DIMM, PCIe, optical connectivity, thermal, and power-related issues.
• Ability to document hardware failures and coordinate physical repair activities.

Scripting & Automation

• Strong troubleshooting skills using Python and Bash.
• Ability to read and understand JSON/YAML configurations.
• Experience analyzing automation failures, workflow logs, and stack traces.

Preferred Qualifications

• Experience supporting large-scale datacenter infrastructure or AI/ML compute environments.
• Exposure to GPU/TPU clusters and high-performance computing infrastructure.
• Experience with DCIM platforms such as NetBox or Foundry.
• Familiarity with distributed systems and large-scale infrastructure automation.
• Experience in 24/7 production operations, SRE, or infrastructure support environments.
• Strong analytical, troubleshooting, documentation, and communication skills.

Benefits at Quadrant Technologies

• Medical, dental, and vision insurance
• Paid time off (PTO)
• Professional development and training opportunities

Quadrant Technologies is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, protected veteran status, age, or any other characteristic protected by applicable federal, state, or local laws