Platform Reliability Engineer

FIRMUS METAL INTERNATIONAL PTE. LTD.; Singapore; On-site; SGD 100,000 - 170,000; Full time

Job summary

Firmus runs large-scale, state-of-the-art AI infrastructure and operates the platform services underpinning the Firmus AI Cloud in production. The Platform Reliability Engineer will handle incidents across Linux, Kubernetes, storage, and virtualization, owning responses from detection to resolution with a focus on automated remediation and runbooks.

The role involves operating the multi-tenant control plane, virtualisation platform, and high-performance storage, ensuring service levels and rapid

Qualifications

• Solid experience operating production infrastructure in a 24/7 environment using site reliability engineering practices.
• Strong working knowledge of Linux systems administration and troubleshooting.
• Working experience with scale-out, parallel or enterprise storage.
• Familiarity with a virtualisation platform (e.g., Proxmox, VMware or KVM).
• Working experience using observability tools.

Responsibilities

• Triage, diagnose and resolve incidents across the stack, owning each incident from escalation to resolution.
• Operate the multi-tenant control and management plane that supports separation between tenant workloads and data.
• Operate the virtualisation platform and shared services to their declared service levels.
• Operate exabyte-scale distributed storage and S3-compatible object storage to service levels, escalating as needed.
• Manage the GPU compute fleet in production and support provisioning and validation activities.
• Contribute to automation by turning fixes into runbooks and automated remediation.
• Diagnose performance issues across the datapath and coordinate with network/storage specialists.
• Deliver changes as code with review, testing, rollout and rollback plans.
• Use the observability stack to track SLIs, query metrics, logs and traces, and improve dashboards.
• Execute patching and vulnerability remediation across services.
• Manage vendor support cases and drive to resolution with evidence.
• Participate in technical recovery during major incidents and blameless post-incident reviews.

Skills

Site reliability engineering

Linux systems administration

Scale-out storage

Virtualisation

Observability

Kubernetes

Tools

Proxmox

VMware

KVM

Ceph

Lustre

GPFS

NetApp

Job description

Role Summary

Firmus runs large-scale,state-of-the-art AI infrastructure built on the latest generation of GPUrack-scale systems and operated as one estate to power the next generation ofAI innovation. The Platform Reliability Engineer is part of the team that operatesthe platform services Firmus AI Cloud runs on in production: the GPU computefleet, exabyte-scale storage, the multi-tenant control and management plane,and the virtualisation and shared services that host it. Together, these arethe foundation every customer workload depends on.

This is a hands-on site reliabilityengineering role that works across the stack, from Linux and Kubernetes tostorage, virtualisation and the compute fleet. The team runs each service to adeclared service level, measures reliability against it, and treats operationsas a software problem, engineering repetitive manual work out of the systemrather than absorbing it. The role picks up incidents escalated from firstresponse, owns each one through to resolution, and works across teams andvendors to restore service, bringing in specialist engineers when a fault needsdeeper investigation. Automation is a first-class part of the role: faults arediagnosed using the observability stack, post-incident reviews feed fixes backinto the platform, and every resolved fault is a candidate for a runbook or anautomated fix.

Key Responsibilities

• Triage, diagnose and resolve incidents acrossthe stack, including Kubernetes clusters and workloads, storage,virtualisation, network connectivity and the underlying Linux hosts, owningeach incident from escalation through to resolution.
• Operate the multi-tenant control and managementplane that Firmus' AI and infrastructure services depend on, including thetenancy, quota and access controls that support separation between tenantworkloads and data.
• Operate the virtualisation platform that hoststhe management plane, and the shared services that run on it, to their declaredservice levels.
• Operate Firmus' exabyte-scale distributed andhigh-performance filesystems and S3-compatible object storage to their declaredservice levels, handling capacity, performance and availability incidents andescalating to specialist engineers and vendors (for example VAST or WEKA) withclear evidence when further investigation is needed.
• Operate the GPU compute fleet in production:node health and readiness, GPU and node fault detection and handling, firmwareand driver currency to the supported baselines, remediation and return toservice, and support for cluster provisioning and validation activities.
• Reduce toil by converting repeat fixes intorunbooks and automated remediation delivered as controlled code, contributingto the team's automation and remediation tooling.
• Diagnose performance problems across the datapath between compute and storage, gathering the evidence needed to resolve themand working with network and storage specialists where required.
• Deliver change as code through the agreed changeprocess, with peer review, testing, staged rollout where applicable, and arollback plan.
• Use the observability stack (for exampleGrafana, Prometheus, Loki and Tempo) to track service level indicators againsttheir objectives, query metrics, logs and traces, correlate signals acrossservices and diagnose faults, and improve dashboards and alerting so that eachalert is actionable and tied to a runbook.
• Execute the patching cycle and urgentvulnerability remediation across the services the team operates.
• Manage vendor support cases across the hardwareand software portfolio, raising and tracking cases under the relevant supportagreements, supplying logs and diagnostic evidence, and following cases throughto agreed response and resolution times.
• Take part in technical recovery during majorincidents and in blameless post-incident reviews, feed recurring issues back tothe engineering teams as clear requirements, share the follow-the-sun on-callroster, and keep incident tickets, operational procedures and runbooks accurateand current so that others can execute them successfully.

Skills & Experience

Required Skills

• Solidexperience operating production infrastructure in a 24/7 environment using sitereliability engineering practices, including service level objectives, toilreduction, owning incidents from escalation to resolution, on-callparticipation, post-incident review and writing runbooks that others canexecute successfully.
• Strongworking knowledge of Linux systems administration and troubleshooting,including storage, host networking, processes, memory and I/O, and a systematicapproach to isolating faults.
• Workingexperience with scale-out, parallel or enterprise storage (for example VAST,WEKA, Ceph, Lustre, GPFS or NetApp), including diagnosing performance andcapacity problems using evidence rather than assumption.
• Familiaritywith a virtualisation platform (for example Proxmox, VMware or KVM).
• Workingexperience using observability t