Observability Lead: Strategy & Telemetry Ownership

H; Greater London; Hybrid; GBP 110,000 - 140,000; Full time

Job summary

H is seeking a hands-on lead to own observability across the full stack, defining the strategy, designing the tech, and guiding teams through rollout. You’ll surface the data that informs key decisions and measures progress.

You will own systems for metrics, logs, traces, alerts and dashboards, build end-to-end visibility of dependencies, and collaborate with product, research and infra teams to define SLAs, SLOs and SLIs.

Qualifications

• Have set the direction for a substantial piece of engineering work, delivered it and got other teams to adopt it.
• Have built production instrumentation and telemetry integrations yourself.
• Have defined both product-level and technical-level SLAs, SLOs and SLIs.
• Have used SRE measurements such as error budgets, burn rates and MTTA/MTTR/MTTM.
• Have built self-managed observability systems, such as OTEL, Prometheus, Grafana, Loki, Mimir, Alloy, Elasticsearch or Kibana.
• Have worked with Kubernetes at scale, enterprise multi-tenancy, inference or asynchronous agent workflows.

Responsibilities

• Setting one observability strategy that can be applied across teams and technical stacks
• Owning the implementation of the systems that manage metrics, logs, traces, alerts and dashboards
• Defining SLAs, SLOs and SLIs with our product, research and infrastructure teams
• Building end-to-end visibility of system dependencies, providing visualisations for users and context for agents
• Automating how we track coverage, reliability, alert quality and the cost of execution

Skills

Leadership

Observability

Instrumentation

SRE metrics

Tools

OTel

Prometheus

Grafana

Loki

Mimir

Elasticsearch

Kibana

Kubernetes

Alloy

Job description

H is seeking a hands-on lead to own observability across the full stack, defining the strategy, designing the tech, and guiding teams through rollout. You’ll surface the data that informs key decisions and measures progress.

You will own systems for metrics, logs, traces, alerts and dashboards, build end-to-end visibility of dependencies, and collaborate with product, research and infra teams to define SLAs, SLOs and SLIs.