Senior AI Infrastructure Engineer, Observability

Fi; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

Firmus Technologies seeks a Senior AI Infrastructure Engineer, Observability, to define and publish health signals for GPU infrastructure used by customers and internal workloads. You will develop dashboards, alerts and runbooks, enabling other teams to operate and recover efficiently.

You will collaborate across commissioning, operations, and telemetry ownership to ensure production-grade signals and fault diagnostics, including XID events, ECC/memory errors, and NVLink issues.

Qualifications

• Bachelor's degree in computer science or a related technical field, or equivalent practical experience.
• 7+ years in GPU, HPC, AI infrastructure or closely related large-scale systems engineering environments, including ownership of health monitoring or diagnostics used by customers or internal teams.
• Experience with production GPU fault diagnosis, including XID events, ECC/memory errors, NVLink issues, power or thermal limits.
• Strong Linux and server fundamentals; ability to debug on the machine and reason across GPU, CPU, memory, PCIe, power and cooling.
• Automation in Python or similar; hands-on with GPU diagnostics and turning them into checks others run.

Responsibilities

• Define GPU and host health criteria and service-readiness gates for repair and capacity workflows.
• Publish golden dashboards, alerts, PromQL/LogQL queries and health checks for adoption.
• Develop diagnostics with DCGM checks, NCCL and bandwidth tests, stress and burn-in for validation.
• Isolate faults by correlating host, cooling, network or power-limit issues using telemetry.
• Collaborate with incident response and operations teams to share knowledge and create reusable checks and runbooks.

Skills

Python scripting

Linux basics

DCGM checks

NCCL tests

PromQL

Grafana dashboards

Incident response

Education

Bachelor's degree in CS or related field

Tools

PromQL

LogQL

Grafana

Job description

Firmus Technologies seeks a Senior AI Infrastructure Engineer, Observability, to define and publish health signals for GPU infrastructure used by customers and internal workloads. You will develop dashboards, alerts and runbooks, enabling other teams to operate and recover efficiently.

You will collaborate across commissioning, operations, and telemetry ownership to ensure production-grade signals and fault diagnostics, including XID events, ECC/memory errors, and NVLink issues.