Principal Engineer, AI Cloud Software

Firmus Technologies Pty Ltd.; Singapore; On-site; SGD 180,000 - 240,000; Full time

Job summary

Firmus Technologies Pty Ltd. seeks a Senior AI Infrastructure Engineer, Observability, to define and measure GPU infrastructure health and service-readiness for customer and internal workloads.

You will publish reusable dashboards, alerts, and runbooks, enabling other teams to operate and recover with clear guidance. Strong Linux, Python, and GPU diagnostics background are essential for success.

Qualifications

• Bachelor's degree in computer science or related field or equivalent practical experience.
• 7+ years in GPU, HPC, AI infrastructure or related large-scale systems engineering environments.
• Experience with production GPU fault diagnosis and health monitoring.
• Strong Linux and server fundamentals; ability to debug across GPU/CPU/memory/PCIe/power/cooling and automate in Python.

Responsibilities

• Define GPU and host health criteria and service-readiness gates for repair and capacity workflows.
• Publish golden dashboards, alerts, PromQL/LogQL queries and health checks for broader adoption.
• Conduct DCGM checks, NCCL tests, stress/burn-in, and validation jobs to verify server performance.
• Diagnose faults using host, BMC telemetry to separate hardware from environmental issues.
• Collaborate with cross-functional teams to escalate and document healthy signals for operations.

Skills

Linux

Python

Diagnostics

Incident response

Education

Bachelor's degree in computer science or related field

Tools

DCGM

NCCL/collective tests

PromQL

LogQL

Grafana

Job description

Firmus Technologies is a global leader pioneering the development and operation of efficient AI infrastructure across Asia Pacific.

Founded in Australia in 2019, our mission is to create the most efficient AI infrastructure by combining cutting-edge technology with a steadfast commitment to sustainability.

At Firmus, we are unique in our approach. We design, build, and operate a new class of digital infrastructure – the AI Factory. Through our model-to-grid technology approach, we have pushed the boundaries of multi-generational liquid cooling systems, energy management, AI software orchestration, and construction. For our customers, this approach allows us to make every watt count and deliver low-cost AI tokens globally.

Firmus AI Cloud

Our large-scale GPU cloud platform, Firmus AI Cloud, is purpose-built to deliver energy-efficient AI compute at scale to customers.

It empowers developers, enterprises, educational institutions, and government users to train and deploy AI models with unmatched efficiency and cost savings. With an ever-growing suite of services and applications, we are committed to delivering a cloud experience that is market-leading, proprietary, and built to scale.

ROLE SUMMARY

Firmus Technologies is seeking a Senior AI Infrastructure Engineer, Observability, to join our Engineering and Technology team. You will establish how we measure, validate and communicate the health of GPU infrastructure used for customer and internal workloads. You will define trusted health signals and service-readiness criteria, and turn them into reusable dashboards, alerts, queries, diagnostic checks and operational guidance. Your work will help commissioning, infrastructure and operations teams bring capacity online safely, identify degradation early and recover from failures quickly. You will also make knowledge self-service by publishing clear reference implementations, runbooks and AI-ready operational knowledge that other teams can use and extend.

KEY RESPONSIBILITIES

• The Health Standard
  Define GPU and host health criteria for customer and internal workloads, and the service-readiness gates that repair and capacity workflows depend on. Cover what stops or corrupts AI jobs: GPUs falling off the bus, XID events, ECC and memory faults, NVLink/NVSwitch degradation, thermal and power capping, NCCL and collective failures, silent data corruption, and stragglers running below fleet baseline.
• Reference Implementations
  Publish golden dashboards, alerts, PromQL/LogQL queries and health checks that other teams adopt and extend. Your output is the standard and the examples. The alerts must be specific, low-noise, and with a clear next action.
  
• Diagnostics with Real Pass/Fail Criteria
  DCGM checks, NCCL and bandwidth tests, stress and burn-in, and validation jobs that confirm a server matches expected performance. Others must be able to run them without you.
• Fault Isolation
  Separate a bad GPU from a cooling, host, network or power-limit problem using host and BMC/management-interface telemetry together, including when the same signature appears across many servers. A clean management view means nothing if the host is throwing faults.
  
• Detection of the Failures that Don't Crash

Rising correctable ECC counts, NVLink retries, thermal slowdown, XID patterns, wrong results with no error. Keep the knowledge current: what each signal means, what to do next on the machine, and where the operation team must make the final decision.

• Partnership and Escalation
  Work with commissioning on bring-up and acceptance, operations on break-fix, infrastructure on what healthy hardware looks like, and the telemetry owners on making your signals production-grade. Run technical sessions with customer teams on the telemetry they need. Join GPU and host incidents, including debugging on the server and convert every finding into a reusable check, alert or runbook. Give engineering leadership a straight read on fleet health and risk.

SKILLS AND EXPERIENCE

• Bachelor's degree in computer science or a related technical field, or equivalent practical experience.
• 7+ years in GPU, HPC, AI infrastructure or closely related large-scale systems engineering environments, including ownership of health monitoring or diagnostics used by customers or internal teams.
• Experience with production GPU fault diagnosis. You've found genuine faults through XID events, ECC/memory errors, NVLink issues, power or thermal limits and caught at least one before the job died.
• Strong Linux and server fundamentals. You can debug on the machine and reason across GPU, CPU, memory, PCIe, power and cooling. You automate in Python or similar.
• Hands-on with GPU diagnostics and validation such as DCGM, NCCL/collective tests, stress testing and you've turned them into checks other people run.
• E xperience analysing infrastructure telemetry, building trusted dashboards and alerts. Savvy with PromQL , LogQL , Grafana or comparable tooling. Able to follow an unexpected signal until there is a cause and can turn that into an alert or check others will trust.
• Uses AI tools as a normal part of analysis and build work. Can structure health knowledge and build AI skills so operation team with AI assistants can use it effectively to recover from incident s .
• Willing to take part in the incident-response on-call rotation.
• Willing to travel overseas occasionally when the role requires it.
• Clear and effective written and verbal communication in English.

Highly Desirable Experiences

• Production experience on large GPU systems with tight GPU-to-GPU interconnect.
• Multiple sites or GPU generations, especially preparing health monitoring before a new GPU class entered production.
• Background at an AI cloud, GPU cloud or HPC centre running customer workloads.
• Experience working with data or ML engineers on GPU/host failure prediction from health signals.
• Built a structured operational knowledge so AI assistants can use it effectively during incident recovery.

Location & Reporting

• Report to: Senior Manager, Platform Engineering

Employment Basis

Full-time

Diversity

At Firmus, we are committed to building a diverse and inclusive workplace. We encourage applications from candidates of all backgrounds who are passionate about creating a more sustainable future through innovative engineering solutions.