HPC Reliability Engineer: Observability & Incident Response

Technology Innovation Institute; Abu Dhabi; Vor Ort; AED 300.000 - 540.000; Vollzeit

Zusammenfassung

Technology Innovation Institute in Abu Dhabi invites a skilled Site Reliability Engineer to fortify the reliability of HPC and GPU clusters. You will design observability, implement alerts, and drive incident response across compute, storage, networking, and orchestration systems.

We seek strong Linux, Kubernetes/Slurm, and Python/Go/Rust scripting abilities, plus experience with Prometheus, Grafana, and centralized logging to deliver timely operational excellence and scalable infrastructure.

Qualifikationen

• Proven experience in site reliability engineering, HPC operations, or infrastructure operations.
• Strong Linux administration and troubleshooting skills for clusters and workloads.
• Experience with Kubernetes or Slurm for workload orchestration.
• Hands-on with observability tools: Prometheus, VictoriaMetrics, Grafana, and centralized logging.
• Ability to write monitoring queries and design actionable alerts.
• Experience investigating failures across compute, GPU, storage, and network systems.
• Proficiency in Python, Bash, Go, or Rust for automation.
• Familiarity with infrastructure-as-code and GitOps practices.
• Strong communication skills and incident management capabilities.
• Experience with GPU telemetry, RDMA networking, or distributed training is a plus.
• Experience operating infrastructure on AWS, GCP, or Azure is a plus.

Aufgaben

• Design, implement, and maintain observability systems across infrastructure and workloads.
• Develop alerts, dashboards, and health checks for HPC clusters and GPU workloads.
• Define service-level indicators and objectives with platform owners.
• Detect, triage, and respond to incidents; coordinate investigation and recovery.
• Participate in on-call rotations and improve runbooks and escalation procedures.
• Diagnose complex failures and performance degradation in GPUs, storage, or networks.
• Conduct root-cause analyses and implement corrective actions to prevent recurrence.
• Develop automation for diagnostics, remediation, and recovery.
• Improve alert quality by reducing noise and identifying monitoring gaps.
• Collaborate with infra engineers, developers, and researchers to boost reliability and capacity planning.

Kenntnisse

Linux administration

Incident detection & response

Telemetry analysis

Python

Go

Rust

Bash

Communication

Root-cause analysis

Capacity planning

Workload performance

Ausbildung

Bachelor's or Master's degree in Computer Science or Electrical Engineering

Tools

Kubernetes

Slurm

Prometheus

VictoriaMetrics

Grafana

Centralized logging systems

GitOps

IaC tooling

AWS

GCP

Azure

Jobbeschreibung

Technology Innovation Institute in Abu Dhabi invites a skilled Site Reliability Engineer to fortify the reliability of HPC and GPU clusters. You will design observability, implement alerts, and drive incident response across compute, storage, networking, and orchestration systems.

We seek strong Linux, Kubernetes/Slurm, and Python/Go/Rust scripting abilities, plus experience with Prometheus, Grafana, and centralized logging to deliver timely operational excellence and scalable infrastructure.