Senior Observability Engineer

Verda; Greater London; Remoto; GBP 100.000 - 140.000; Tempo pieno

Vantaggi offerti da questo lavoro

Equity compensation

Local benefits

Descrizione del lavoro

Verda is building a distributed AI cloud spanning data centers, hardware and a scalable cloud platform used by leading AI teams.

We’re hiring a Senior Observability Engineer to help design and operate the observability platform across bare-metal, VM, Linux, Kubernetes, GPU clusters and cloud-native apps.

Competenze

• Experience with Grafana and VictoriaMetrics in production telemetry.
• Strong Linux fundamentals and production Kubernetes experience.
• Experience with OpenTelemetry instrumentation and Collector pipelines.

Mansioni

• Design, scale, and operate shared metrics, logs, and tracing services.
• Manage declarative Kubernetes deployments using Argo CD and GitLab/GitHub CI/CD.
• Improve monitoring for bare-metal hardware, Linux, networks, and Kubernetes.
• Build telemetry pipelines for GPU clusters, AI training runs, and inference services.
• Define and promote shared OpenTelemetry standards across teams.
• Improve alerting, incident response, and reduce noise.

Conoscenze

Grafana

VictoriaMetrics

OpenTelemetry

Kubernetes

Linux fundamentals

CI/CD

GitOps

Ansible

SaltStack

Instrumentation (Go)

Strumenti

Argo CD

GitLab CI/CD

GitHub CI/CD

Prometheus

OpenTelemetry

Grafana

VictoriaLogs

Descrizione del lavoro

At Verda, we’re building a full-stack AI cloud, covering everything from data centers and hardware to our own cloud platform that the world’s leading AI teams use to do serious AI work.

We strive to make a positive mark on the world through the infrastructure we build and give leading teams a service they can truly depend on. Headquartered in Helsinki, we operate globally with offices in London and San Francisco.

Join Verda while it’s still being built - not once it’s finished.

Why Verda

• Cash and equity compensation along with local benefits.
• 40+ nationalities, with 6 different ones on the management team.
• A real chance to make an impact and work alongside world class engineers, researchers, and partners across the global AI ecosystem.

About The Role

As a Senior Observability Engineer on our Observability Platform team, you will contribute to the architecture and development of our observability platform across bare-metal servers, virtual machines, Linux, Kubernetes, GPU clusters, network infrastructure, and cloud-native applications. You will help translate architectural decisions into practical implementations and shared standards. A core part of your work will involve operating and improving our observability stack built on Grafana and tools from the VictoriaMetrics and Prometheus ecosystems, improving incident detection and response, and developing AI-assisted monitoring capabilities. Working closely with platform, SRE, infrastructure, network, security, application, and ML engineering teams, you will help integrate observability into how systems are designed and operated.

Your responsibilities

• Observability platform: Design, scale, and operate shared metrics, logs, and tracing services, covering infrastructure health and workload behavior.
• GitOps & automation: Manage declarative Kubernetes deployments using Argo CD and GitLab and GitHub CI/CD, and automate host and VM configuration using Ansible and SaltStack.
• Infrastructure telemetry: Improve monitoring for bare-metal hardware, Linux, physical and overlay networks, and Kubernetes. Integrate telemetry from sources such as IPMI/Redfish, PDUs, SNMP, gNMI, and NetFlow/sFlow.
• GPU & AI workloads: Build telemetry pipelines and dashboards for GPU clusters, AI training runs, and inference services.
• Application telemetry: Help define shared standards for OpenTelemetry instrumentation, context propagation, and Collector pipelines. Support teams instrumenting Go, Node.js, and Python services and APIs.
• Alerting & incident response: Build useful dashboards and alerts, improve notification workflows through PagerDuty, and participate in on-call rotations. Work with other teams to reduce alert noise and shorten incident diagnosis and recovery.
• Shared practices: Document standards, help teams adopt them, and evaluate and develop AI-assisted approaches to monitoring and incident investigation.

Your key competencies

• Observability platforms: Strong hands-on experience with Grafana and production telemetry systems, including VictoriaMetrics and VictoriaLogs or comparable metrics and logging backends.
• Infrastructure & distributed systems: Strong Linux fundamentals, production Kubernetes experience, and experience monitoring cloud-native applications and distributed systems.
• OpenTelemetry: Practical experience with instrumentation, metrics, traces, structured logs, trace context propagation, and Collector pipelines.
• GitOps & automation: Experience with declarative deployments, CI/CD, and configuration management using tools such as Argo CD, GitLab CI/CD, Ansible, and SaltStack.
• Operations: Experience investigating production incidents, maintaining alerts, and sharing operational responsibilities within an engineering team.
• Cross-team collaboration: Ability to explain technical decisions, guide teams, and support adoption of shared observability practices.

Nice to have

• Experience building and operating large-scale observability systems across multiple data centers.
• Experience monitoring GPU clusters and AI/ML workloads, including NVIDIA DCGM and training or inference performance metrics.
• Deeper experience with hardware and network telemetry, such as IPMI/Redfish, NetFlow/sFlow, SNMP, gNMI, or eBPF.
• Experience operating across bare-metal, VM, and Kubernetes environments.
• Experience applying LLMs, ML, or agent-based systems to automation and incident investigation.Exposure to security monitoring and collaboration with SecOps teams.

Practicalities

• Location: London (UK), Helsinki (Finland), or remote in EU
• Employment type: Full time and permanent