Senior Observability Engineer

Hub; Greater London; Remote; GBP 90,000 - 140,000; Full time

Job summary

Verda is building a full-stack AI cloud, enabling world‑leading teams to do serious AI work from data centers to cloud platforms. We seek a Senior Observability Engineer to architect and operate our observability stack across bare‑metal, VM, Linux, Kubernetes, GPU clusters, and cloud-native apps.

You will define instrumentation standards, build AI-assisted monitoring, and guide cross‑team adoption while contributing to incident detection and response. London or remote in EU options are available.

Qualifications

• Strong hands-on experience with Grafana and production telemetry systems (VictoriaMetrics/Logs).
• Strong Linux fundamentals, production Kubernetes experience and cloud-native monitoring.
• Practical OpenTelemetry instrumentation, metrics, traces, logs and collectors.
• Experience with declarative deployments, CI/CD and configuration management (Argo CD, GitLab CI/CD, Ansible, SaltStack).
• Experience investigating production incidents, maintaining alerts, and sharing ops responsibilities.
• Ability to explain technical decisions and drive adoption of shared observability practices.

Responsibilities

• Design, scale, and operate shared metrics, logs, and tracing services.
• Manage deployments with Argo CD, GitLab CI/CD, and automate host/VM configuration.
• Improve monitoring for bare-metal, Linux, networks, and Kubernetes with telemetry sources.
• Build telemetry pipelines and dashboards for GPU clusters, AI runs, and inference services.
• Define OpenTelemetry instrumentation standards for Go, Node.js and Python services.
• Develop dashboards and alerts; participate in on-call rotations and improve incident response.
• Document standards and guide teams in adopting observability practices.

Skills

Grafana

VictoriaMetrics

Linux basics

Kubernetes

OpenTelemetry

GitOps

Automation

Incident response

Cross‑team collab

Tools

Argo CD

GitLab CI/CD

Ansible

SaltStack

Job description

At Verda, we're building a full-stack AI cloud, covering everything from data centers and hardware to our own cloud platform that the world's leading AI teams use to do serious AI work.

We strive to make a positive mark on the world through the infrastructure we build and give leading teams a service they can truly depend on. Headquartered in Helsinki, we operate globally with offices in London and San Francisco.

Join Verda while it's still being built - not once it's finished.

About the role

As a Senior Observability Engineer on our Observability Platform team, you will contribute to the architecture and development of our observability platform across bare-metal servers, virtual machines, Linux, Kubernetes, GPU clusters, network infrastructure, and cloud-native applications.

You will help translate architectural decisions into practical implementations and shared standards. A core part of your work will involve operating and improving our observability stack built on Grafana and tools from the VictoriaMetrics and Prometheus ecosystems, improving incident detection and response, and developing AI-assisted monitoring capabilities.

Working closely with platform, SRE, infrastructure, network, security, application, and ML engineering teams, you will help integrate observability into how systems are designed and operated.

Your responsibilities

• Observability platform: Design, scale, and operate shared metrics, logs, and tracing services, covering infrastructure health and workload behavior.
• GitOps & automation: Manage declarative Kubernetes deployments using Argo CD and GitLab and Github CI/CD, and automate host and VM configuration using Ansible and SaltStack.
• Infrastructure telemetry: Improve monitoring for bare-metal hardware, Linux, physical and overlay networks, and Kubernetes. Integrate telemetry from sources such as IPMI/Redfish, PDUs, SNMP, gNMI, and NetFlow/sFlow.
• GPU & AI workloads: Build telemetry pipelines and dashboards for GPU clusters, AI training runs, and inference services.
• Application telemetry: Help define shared standards for OpenTelemetry instrumentation, context propagation, and Collector pipelines. Support teams instrumenting Go, Node.js, and Python services and APIs.
• Alerting & incident response: Build useful dashboards and alerts, improve notification workflows through PagerDuty, and participate in on‑call rotations. Work with other teams to reduce alert noise and shorten incident diagnosis and recovery.
• Shared practices: Document standards, help teams adopt them, and evaluate and develop AI‑assisted approaches to monitoring and incident investigation.

Your key competencies

• Observability platforms: Strong hands‑on experience with Grafana and production telemetry systems, including VictoriaMetrics and VictoriaLogs or comparable metrics and logging backends.
• Infrastructure & distributed systems: Strong Linux fundamentals, production Kubernetes experience, and experience monitoring cloud‑native applications and distributed systems.
• OpenTelemetry: Practical experience with instrumentation, metrics, traces, structured logs, trace context propagation, and Collector pipelines.
• GitOps & automation: Experience with declarative deployments, CI/CD, and configuration management using tools such as Argo CD, GitLab CI/CD, Ansible, and SaltStack.
• Operations: Experience investigating production incidents, maintaining alerts, and sharing operational responsibilities within an engineering team.
• Cross‑team collaboration: Ability to explain technical decisions, guide teams, and support adoption of shared observability practices.

Nice to have

• Experience building and operating large‑scale observability systems across multiple data centers.
• Experience monitoring GPU clusters and AI/ML workloads, including NVIDIA DCGM and training or inference performance metrics.
• Deeper experience with hardware and network telemetry, such as IPMI/Redfish, NetFlow/sFlow, SNMP, gNMI, or eBPF.
• Experience operating across bare‑metal, VM, and Kubernetes environments.
• Experience applying LLMs, ML, or agent‑based systems to automation and incident investigation.
• Exposure to security monitoring and collaboration with SecOps teams.

Why Verda

• Cash and equity compensation along with local benefits.
• 40+ nationalities, with 6 different ones on the management team.
• A real chance to make an impact and work alongside world class engineers, researchers, and partners across the global AI ecosystem.

Practicalities

• Location: London, UK or remote in EU
• Employment type: Full time and permanent

What's next

We're building fast and this role needs the right person behind it. There's no artificial deadline, but when we find who we're looking for, we move.