Site Reliability Engineer — HPC Observability And Incident Response

Technology Innovation Institute; Abu Dhabi; Vor Ort; AED 300.000 - 540.000; Vollzeit

Zusammenfassung

Technology Innovation Institute in Abu Dhabi invites a skilled Site Reliability Engineer to fortify the reliability of HPC and GPU clusters. You will design observability, implement alerts, and drive incident response across compute, storage, networking, and orchestration systems.

We seek strong Linux, Kubernetes/Slurm, and Python/Go/Rust scripting abilities, plus experience with Prometheus, Grafana, and centralized logging to deliver timely operational excellence and scalable infrastructure.

Qualifikationen

• Proven experience in site reliability engineering, HPC operations, or infrastructure operations.
• Strong Linux administration and troubleshooting skills for clusters and workloads.
• Experience with Kubernetes or Slurm for workload orchestration.
• Hands-on with observability tools: Prometheus, VictoriaMetrics, Grafana, and centralized logging.
• Ability to write monitoring queries and design actionable alerts.
• Experience investigating failures across compute, GPU, storage, and network systems.
• Proficiency in Python, Bash, Go, or Rust for automation.
• Familiarity with infrastructure-as-code and GitOps practices.
• Strong communication skills and incident management capabilities.
• Experience with GPU telemetry, RDMA networking, or distributed training is a plus.
• Experience operating infrastructure on AWS, GCP, or Azure is a plus.

Aufgaben

• Design, implement, and maintain observability systems across infrastructure and workloads.
• Develop alerts, dashboards, and health checks for HPC clusters and GPU workloads.
• Define service-level indicators and objectives with platform owners.
• Detect, triage, and respond to incidents; coordinate investigation and recovery.
• Participate in on-call rotations and improve runbooks and escalation procedures.
• Diagnose complex failures and performance degradation in GPUs, storage, or networks.
• Conduct root-cause analyses and implement corrective actions to prevent recurrence.
• Develop automation for diagnostics, remediation, and recovery.
• Improve alert quality by reducing noise and identifying monitoring gaps.
• Collaborate with infra engineers, developers, and researchers to boost reliability and capacity planning.

Kenntnisse

Linux administration

Incident detection & response

Telemetry analysis

Python

Go

Rust

Bash

Communication

Root-cause analysis

Capacity planning

Workload performance

Ausbildung

Bachelor's or Master's degree in Computer Science or Electrical Engineering

Tools

Kubernetes

Slurm

Prometheus

VictoriaMetrics

Grafana

Centralized logging systems

GitOps

IaC tooling

AWS

GCP

Azure

Jobbeschreibung

Job Description:

The Institute

Technology Innovation Institute (TII) a publicly funded research institute based in Abu Dhabi, the United Arab Emirates. It is home to a diverse community of leading scientists, engineers, mathematicians, and researchers from across the globe, transforming problems and roadblocks into pioneering research and technology prototypes that help move society ahead.

AI and Digital Science Research Centre

The Artificial Intelligence and Digital Science Research Centre (AIDRC) is the machine learning powerhouse of TII. We work closely with other research centers to harness the full potential of AI across multiple domains — from healthcare and cybersecurity to government and advanced technology systems.

We're looking for a talented Site Reliability Engineer with expertise in HPC and observability to join our growing team and help maintain the reliability of our computational infrastructure. As an engineer in our team, you'll develop monitoring and incident response capabilities for large-scale CPU and GPU clusters, detect operational issues early, and drive their resolution across compute, storage, networking, and workload orchestration systems.

Responsibilities

• Design, implement, and maintain observability systems covering infrastructure and workloads through metrics, logs, and traces.
• Develop actionable alerts, dashboards, and health checks for HPC clusters, Kubernetes platforms, GPU workloads, storage, and networking.
• Define service-level indicators and objectives in collaboration with platform owners and research teams.
• Detect, triage, and respond to incidents, coordinating investigation, escalation, mitigation, and service recovery.
• Participate in an on-call rotation and improve operational readiness through runbooks, escalation procedures, and incident exercises.
• Diagnose complex failures and performance degradation, including GPU errors, scheduling issues, storage bottlenecks, and network faults.
• Conduct root-cause analysis and blameless incident reviews, and implement corrective actions to prevent recurrence.
• Develop automation for incident enrichment, diagnostics, remediation, and recovery.
• Improve alert quality by reducing noise, identifying monitoring gaps, and detecting persistent resource underutilization or abnormal workload behavior.
• Collaborate with infrastructure engineers, developers, and researchers to improve reliability, capacity planning, and workload performance.

Requirements

• Proven experience in site reliability engineering, HPC operations, or infrastructure operations, with responsibility for incident detection and response.
• Strong Linux administration and troubleshooting skills, including processes, memory, filesystems, networking, and cgroups.
• Experience operating clustered CPU/GPU infrastructure and workload orchestration platforms such as Kubernetes or Slurm.
• Hands‑on experience with observability tools such as Prometheus, VictoriaMetrics, Grafana, and centralized logging systems.
• Ability to write monitoring queries, investigate telemetry, and design alerts that support timely operational decisions.
• Experience investigating failures across compute, GPU, storage, and network systems.
• Proficiency in Python, Bash, Go, Rust, or another language used for operational automation.
• Familiarity with infrastructure-as-code, configuration management, and GitOps practices.
• Understanding of incident management, escalation, service recovery, and root‑cause analysis.
• Strong analytical and communication skills, including the ability to communicate clearly during incidents.
• Experience with GPU telemetry, high-performance storage, RDMA networking, distributed training, or parallel computing frameworks is a plus.
• Experience operating infrastructure on AWS, GCP, or Azure is a plus.

Qualifications

• Bachelor's or Master's degree in Computer Science, Electrical Engineering, or a related field.