Senior AI Infra Observability Engineer

Firmus Technologies Pty Ltd.; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

Firmus Technologies is seeking a Senior AI Infrastructure Engineer, Observability, to lead the health measurement and readiness criteria for GPU infrastructure used in customer and internal workloads.

You will define trusted health signals, publish dashboards and alerts, and produce actionable guidance to help commissioning, operations, and incident teams respond quickly and safely. Strong Linux, Python and GPU diagnostics are essential.

Qualifications

• Bachelor’s degree in computer science or a related technical field, or equivalent practical experience.
• 7+ years in GPU, HPC, AI infrastructure or closely related large-scale systems engineering environments with health monitoring ownership.
• Experience with production GPU fault diagnosis and identifying faults via XID, ECC/memory, NVLink, power or thermal limits.
• Strong Linux/server fundamentals; ability to debug on machine and reason across GPU/CPU/memory/PCIe/power/cooling.
• Hands-on GPU diagnostics and validation; experience turning checks into repeatable runs.
• Experience building dashboards and alerts with PromQL/LogQL; capable of reliable, production-grade telemetry.
• Willing to participate in incident-response on-call rotations and travel occasionally.
• Clear, effective English communication.

Responsibilities

• Define GPU and host health criteria and service-readiness gates for workloads and capacity workflows.
• Publish reusable dashboards, alerts, and health checks; ensure outputs are low-noise and actionable.
• Develop diagnostics with pass/fail criteria; create runbooks and reference implementations that others can run.
• Isolate faults by correlating telemetry across host, GPU, and management interfaces.
• Maintain and update health signals as new hardware and AI workloads evolve.
• Collaborate with incident response, operations, telemetry owners, and customers to improve reliability.

Skills

Health monitoring

Linux

Python

DCGM

NCCL

PromQL

LogQL

Grafana

GPU diagnostics

On-call rotation

Education

Bachelor’s degree in Computer Science or related field

Tools

DCGM

NCCL/collective tests

NVIDIA NVLink/NVSwitch

PromQL

Grafana

Job description

Firmus Technologies is seeking a Senior AI Infrastructure Engineer, Observability, to lead the health measurement and readiness criteria for GPU infrastructure used in customer and internal workloads.

You will define trusted health signals, publish dashboards and alerts, and produce actionable guidance to help commissioning, operations, and incident teams respond quickly and safely. Strong Linux, Python and GPU diagnostics are essential.