Senior AI Infra Engineer, Observability

Firmus Technologies; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

Firmus Technologies in Singapore seeks a Senior AI Infrastructure Engineer, Observability, to define health signals and service-readiness criteria for GPU infrastructure powering customer and internal workloads.

You will publish dashboards, alerts, and runbooks, enable self-service knowledge, and work with operations, engineering, and telemetry teams to keep fleet healthy and prepare for future AI workloads.

Qualifications

• Bachelor's degree in computer science or a related technical field, or equivalent practical experience.
• 7+ years in GPU, HPC, AI infrastructure or related large-scale systems engineering environments with health monitoring ownership.
• Experience with production GPU fault diagnosis involving XID, ECC/memory errors, NVLink issues, power or thermal limits.
• Strong Linux and server fundamentals; ability to debug on the machine and reason across GPU/CPU/memory/PCIe/power/cooling.
• Hands-on with GPU diagnostics and validation (DCGM, NCCL tests, stress tests) and converting them to checks for others.
• Experience analysing infrastructure telemetry and building trusted dashboards and alerts (PromQL, LogQL, Grafana).
• Uses AI tools in analysis and has prepared operational knowledge for AI assistants.

Responsibilities

• Define GPU and host health criteria and service-readiness gates for repair and capacity workflows.
• Publish golden dashboards, alerts, queries and health checks for adoption by other teams.
• Run DCGM checks, NCCL tests, stress/burn-in and validation jobs to confirm server performance.
• Isolate faults by using telemetry to distinguish hardware, cooling, network or power-limit issues.
• Maintain knowledge of signals, what they mean, and actions for incident recovery.

Skills

Health monitoring

GPU diagnostics

Linux fundamentals

Python automation

PromQL/LogQL

Telemetry dashboards

Education

Bachelor's degree in CS or related field

Tools

DCGM

NCCL/Tests

PromQL/LogQL

Grafana

Job description

Firmus Technologies in Singapore seeks a Senior AI Infrastructure Engineer, Observability, to define health signals and service-readiness criteria for GPU infrastructure powering customer and internal workloads.

You will publish dashboards, alerts, and runbooks, enable self-service knowledge, and work with operations, engineering, and telemetry teams to keep fleet healthy and prepare for future AI workloads.