Senior AI Cloud Infra Observability Engineer

Firmus Technologies Pty Ltd.; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

Firmus Technologies Pty Ltd. seeks a Senior AI Infrastructure Engineer, Observability, to define and measure GPU infrastructure health and service-readiness for customer and internal workloads.

You will publish reusable dashboards, alerts, and runbooks, enabling other teams to operate and recover with clear guidance. Strong Linux, Python, and GPU diagnostics background are essential for success.

Qualifications

• Bachelor's degree in computer science or related field or equivalent practical experience.
• 7+ years in GPU, HPC, AI infrastructure or related large-scale systems engineering environments.
• Experience with production GPU fault diagnosis and health monitoring.
• Strong Linux and server fundamentals; ability to debug across GPU/CPU/memory/PCIe/power/cooling and automate in Python.

Responsibilities

• Define GPU and host health criteria and service-readiness gates for repair and capacity workflows.
• Publish golden dashboards, alerts, PromQL/LogQL queries and health checks for broader adoption.
• Conduct DCGM checks, NCCL tests, stress/burn-in, and validation jobs to verify server performance.
• Diagnose faults using host, BMC telemetry to separate hardware from environmental issues.
• Collaborate with cross-functional teams to escalate and document healthy signals for operations.

Skills

Linux

Python

Diagnostics

Incident response

Education

Bachelor's degree in computer science or related field

Tools

DCGM

NCCL/collective tests

PromQL

LogQL

Grafana

Job description

Firmus Technologies Pty Ltd. seeks a Senior AI Infrastructure Engineer, Observability, to define and measure GPU infrastructure health and service-readiness for customer and internal workloads.

You will publish reusable dashboards, alerts, and runbooks, enabling other teams to operate and recover with clear guidance. Strong Linux, Python, and GPU diagnostics background are essential for success.