Sr. Observability Systems Engineer

WebHosting; United States; Remote; USD 178,000 - 200,000; Full time

Job summary

Backblaze is seeking a Remote, full-time Senior Infrastructure leader to own architectural direction for observability across data centers. You will guide metrics, logs, and traces, mentor teams, and drive automation with AI-powered approaches for scalable infrastructure.

This senior IC role requires deep distributed systems and cross-team collaboration. You will own metrics pipelines (VictoriaMetrics, Prometheus), logging (OpenSearch, Logstash, Filebeat), tracing (Jaeger, OpenTelemetry), and

Qualifications

• Senior-level observability and infra ownership.
• Experience shaping metrics, logs, traces, and dashboards across multi-cluster environments.
• Strong collaboration with SRE, Storage, and Network teams.

Responsibilities

• Set technical direction for observability across the organization, including instrumentation, alerts, and dashboards.
• Mentor engineers on the Observability team and act as a resource for several engineering domains.
• Lead design reviews for observability work and represent the team in cross-functional discussions.
• Evaluate build-vs-buy decisions for observability and AIOps tooling, including data sovereignty and compliance.
• Produce and maintain design docs, runbooks, and ARs to keep the platform operable and extensible.
• Own and evolve metrics pipelines, logging stacks, and distributed tracing across data centers.
• Define Grafana standards and drive telemetry expansion with product teams.
• Define capacity and retention strategies for telemetry as the fleet grows.
• Own IaC for observability across multiple Kubernetes clusters.
• Write reviews for automation across Debian-based bare metal servers.
• Collaborate with Network Eng on BGP and network-level observability.

Skills

Observability

SRE collaboration

Distributed systems

Tools

VictoriaMetrics

Prometheus

OpenTelemetry

Jaeger

Grafana

OpenSearch

Logstash

Filebeat

Consul

Kubernetes

Ceph

BGP

Job description

Remote Full-time Senior Infrastructure

Job Description

Role Overview

This is a senior individual contributor role with significant architectural ownership. You will set the technical direction for how Backblaze detects, correlates, and responds to infrastructure incidents across our data centers, and you will mentor other engineers on the team as that platform matures. If you have a passion for making complex distributed systems visible, you are comfortable making architecture decisions that will outlive your tenure, and you are excited about applying AI and agentic automation to infrastructure operations, this role was built for you..

Key Responsibilities

Technical Leadership

• Set technical direction and standards for observability across the organization, including instrumentation conventions, alerting philosophy, and dashboard design patterns.
• Mentor engineers on the Observability team and act as a technical resource for SRE, Storage Engineering, and Network Engineering on observability practice.
• Lead design reviews for observability work and represent the team in cross-functional architecture discussions.
• Evaluate build-versus-buy decisions for observability and AIOps tooling, including data sovereignty and compliance considerations.
• Produce and maintain design documentation, runbooks, and architectural decision records that make the platform operable and extensible by future contributors.

Observability Platform

• Own and evolve our metrics pipeline, including VictoriaMetrics, Prometheus, and the Consul-based service discovery layer that keeps scrape targets current as infrastructure scales.
• Own and expand our logging stack (OpenSearch, Logstash, and Filebeat) across all Backblaze data centers.
• Architect and expand distributed tracing with Jaeger and OpenTelemetry, extending coverage beyond the current application instrumentation to new services.
• Define and maintain Grafana dashboard standards for infrastructure, storage, and application performance visibility.
• Drive expansion of internal application telemetry in partnership with application teams to improve visibility.
• Define capacity and retention strategy for the telemetry stack as the fleet grows.

Infrastructure & Automation

• Own infrastructure-as-code for observability stack deployments across multiple Kubernetes clusters.
• Write and review automated playbooks for configuration management and automation across Debian-based bare metal servers.
• Partner with the SRE team on Consul-based service discovery for VictoriaMetrics scrape target automation.
• Lead Ceph storage observability, including OSD health, performance metrics, and integration with alerting.
• Collaborate with the Network Engineering team on BGP, routing, and network-level observability.

Preferred Qualifications

What You’ll Work On

Pay Transparency

To provide greater transparency to candidates, we share base pay ranges for all US-based job postings regardless of state. Final offer amounts are determined by multiple factors including candidate location, skills, depth of work experience, and relevant licenses/credentials.

The expected salary range for this role is $178,000 – $200,000 plus 10% target annual bonus and RSUs.

At Backblaze, we value being fair and good to our customers, partners, and employees. Diversity, equity, and inclusion are at the core of our values. We are committed to fostering a workforce where all employees feel a sense of belonging regardless of race, ethnicity, nationality, gender, sexual orientation, age, religion, socio-economic status, ability, veteran status, and education. We are proud to be an Equal Opportunity Employer.