Staff Observability Platform Engineer: Scale Metrics & Telemetry

Programming.com; United States; Hybrid; USD 180,000 - 240,000; Full time

Job summary

Programming.com seeks a deeply hands-on Staff Observability Platform Engineer to design, build, and operate large-scale metrics, logging, and telemetry infrastructure in a hybrid Kubernetes environment. You will own the observability backend across multiple backends and drive decisions on cardin ality, retention, and cost, collaborating with platform, infra, security, and application teams.

The role demands engineering telemetry at scale, performance-minded storage and query design, and a track

Qualifications

• Operate a metrics/logs backend at production scale and quantify scale.
• Experience with Prometheus-based architectures and remote write.
• Ability to explain architectural decisions around cardinality, retention, and cost.
• Hands-on Kubernetes experience at scale and strong code-reading skills.

Responsibilities

• Design, build, operate, and scale production metrics, logging, tracing, and telemetry platforms.
• Own observability backend architecture supporting large distributed Kubernetes environments.
• Operate and scale backends like Mimir, Thanos, VictoriaMetrics, Cortex, Loki, Elasticsearch.
• Design Prometheus-based architectures including remote write, ingestion pipelines, high availability, retention, and global querying.
• Engineer telemetry systems for growth in data volumes.
• Identify and remediate high-cardinality metrics.
• Make trade-offs across cardinality, retention, storage, ingestion, query performance, reliability, and cost.
• Build and operate OpenTelemetry Collector pipelines.
• Establish observability standards across engineering teams.
• Troubleshoot production issues spanning metrics, logs, traces, Kubernetes, networking, and storage.
• Write and review production-quality code with security and reliability.
• Partner with platform, infrastructure, security, and application engineering teams.

Skills

Hands-on SRE experience

Observability design

Distributed systems problem-solving

Security and reliability focus

Tools

Go

Python

Kubernetes

OpenTelemetry

Prometheus

Loki

Thanos

Mimir

VictoriaMetrics

Cortex

Elasticsearch

Job description

Programming.com seeks a deeply hands-on Staff Observability Platform Engineer to design, build, and operate large-scale metrics, logging, and telemetry infrastructure in a hybrid Kubernetes environment. You will own the observability backend across multiple backends and drive decisions on cardin ality, retention, and cost, collaborating with platform, infra, security, and application teams.

The role demands engineering telemetry at scale, performance-minded storage and query design, and a track