Principal Observability Engineer - AI Infra Telemetry

Firmus Technologies Pty Ltd.; Singapore; Hybrid; SGD 180,000 - 300,000; Full time

Job summary

Firmus Technologies is seeking a Principal Engineer, Observability to own how the AI factory is observed across data centres and cloud regions. You will define telemetry standards, build and scale the platform, and empower teams to own dashboards and alerts from templates.

The role requires 10+ years in software or platform engineering, with hands-on work in Go or Python, and experience with Prometheus/OpenTelemetry in a large-scale environment.

Qualifications

• Bachelor’s degree in computer science or related field.
• 10+ years in software, infrastructure, or platform engineering, incl. 3+ years at principal level.
• Designed a metrics, logs, and traces platform handling millions of samples per second.
• Experience with telemetry ingestion layers (Kafka/Redpanda) and OpenTelemetry.
• Strong Go or Python production services and APIs.

Responsibilities

• Own the architecture of the observability platform across data centres and cloud regions.
• Write production code for telemetry collection, query performance, and observability APIs.
• Make observability self-service for teams with dashboards, alerts, and SLOs as code.
• Set delivery guarantees and data retention policies; manage cardinality and retention.
• Develop live topology model (digital twin) of sites and inventory links.

Skills

Go

Python

Kubernetes

Prometheus

OpenTelemetry

Distributed systems

Telemetry

English communication

Education

Bachelor’s degree in computer science or related field

Tools

VictoriaMetrics

Thanos

Mimir

Kafka

OpenTelemetry Collector

Redfish

gNMI/OpenConfig

Job description

Firmus Technologies is seeking a Principal Engineer, Observability to own how the AI factory is observed across data centres and cloud regions. You will define telemetry standards, build and scale the platform, and empower teams to own dashboards and alerts from templates.

The role requires 10+ years in software or platform engineering, with hands-on work in Go or Python, and experience with Prometheus/OpenTelemetry in a large-scale environment.