Senior Observability Engineer (GCP & Open Telemetry)

Scalence; Morristown (NJ); On-site; USD 120,000 - 160,000; Full time

Job summary

Scalence is seeking a Senior Observability Engineer (GCP & OpenTelemetry) to lead the design and implementation of a telemetry pipeline. You will curate a metrics environment by identifying key signals and implementing label enrichment strategies for data usefulness and cost-effectiveness.

The ideal candidate will have extensive knowledge of Google Cloud Platform resources, OpenTelemetry configuration, and will use infrastructure as code tools like Terraform to ensure observability standards.

Qualifications

• Deep understanding of GCP resource hierarchies and Cloud Monitoring API.
• Advanced configuration of the otel-collector-contrib distribution for monitoring.
• Knowledge of Golden Signals relevant for GCP services.

Responsibilities

• Design, deploy, and maintain OTel Collectors across GKE and GCE.
• Configure receivers, processors, and exporters for data collection.
• Distinguish between noise and signal in GCP metrics.

Skills

GCP Expertise

OpenTelemetry Proficiency

Metric Strategy

Contextual Enrichment

Infrastructure as Code

Tools

OpenTelemetry (OTel) Collector Contrib

Google Cloud Observability (Stackdriver)

Kubernetes (GKE) & Helm

Terraform / Terragrunt

PromQL & MQL

Job description

Senior Observability Engineer (GCP & Open Telemetry)

Senior Observability Engineer (GCP & OpenTelemetry)

We are looking for an Observability specialist to lead the design and implementation of our telemetry pipeline using OpenTelemetry (OTel) Collectors to monitor our GCP infrastructure. You won’t just “turn on ” monitoring; you will curate a high-signal environment by identifying “value-add ” metrics and implementing sophisticated label enrichment strategies to ensure our data is actionable, cost-effective, and context-rich.

KEY RESPONSIBILITIES

• OTel Collector Architecture: Design, deploy, and maintain OTel Collectors (Sidecars, DaemonSets, and Gateway clusters) across GKE and GCE environments.
• Pipeline Optimization: Configure receivers (Google Cloud Monitoring, OTLP, Host Metrics), processors (Batch, Memory Limiter, Resource Detection), and exporters (GoogleCloud, Prometheus).
• GCP Metric Curation: Distinguish between “noise ” and “signal ” by identifying and collecting high-value GCP metrics (e.g., compute.googleapis.com/instance/cpu/scheduler_wait_time vs. simple utilization).
• Metadata & Label Enrichment: Use the Resource Detection Processor and Transform Processor to automatically inject GCP-specific metadata (Project ID, Zone, Instance ID, Custom Labels) into all telemetry signals.
• Cost Management: Implement filtering and dropped-label strategies to manage “cardinality explosions ” and optimize Google Cloud Observability (Stackdriver) costs.

TECHNICAL SKILLS REQUIRED

• GCP Expertise: Deep understanding of GCP resource hierarchies and the Cloud Monitoring API (v3).
• OpenTelemetry Proficiency: Advanced configuration of the otel-collector-contrib distribution, specifically for infrastructure monitoring.
• Metric Strategy: Knowledge of which “Golden Signals ” (Latency, Errors, Saturation, Traffic) are most relevant for specific GCP services like Cloud SQL, GKE, and Pub/Sub.
• Contextual Enrichment: Experience using OTel to bridge the gap between infra metrics and application context (e.g., mapping a GCE instance ID to a specific Business Unit via labels).
• Infrastructure as Code: Proficiency in Terraform or Helm for deploying observability as a standard part of the landing zone.

THE IDEAL CANDIDATE

We’re mostly looking for a Data Curator rather than just a “Systems Admin. ” Most people can install a collector; very few know how to make the data coming out of it actually useful and cost-efficient.
We need an Observability Engineer who acts like a filter. They should know which specific GCP metrics actually matter (so we don’t drown in noise) and how to ‘tag’ (enrich) those metrics using OpenTelemetry so that when an alert goes off, we know exactly which team, project, and environment it belongs to.

Keep an eye out for these on a resume or listen for them in a screening:

• “The Optimizer “: They talk about Cost Management or Cardinality.
  • Why: Storing every single metric in GCP is expensive. A good candidate mentions “filtering ” or “dropping ” useless metrics to save money.
• “The Context King “: They mention Resource Detection or Attribute Mapping.
  • Why: This is the “label enrichment ” part. It means they know how to automatically attach metadata (like Owner: Payments-Team ) to a raw metric.
• “The Contribution Pro “: They mention using the “Contrib ” version of the OTel Collector.
  • Why: The standard OTel collector is basic; the “Contrib ” version contains the specific Google Cloud processors needed for high-level monitoring.

TOP 5 TECH STACK NEEDS

• OpenTelemetry (OTel) Collector Contrib: This is the primary engine where the “magic ” happens; it contains the specific processors required to automatically enrich metrics with GCP metadata and transform raw data into businessready signals.
• Google Cloud Observability (Stackdriver): The candidate must deeply understand the destination’s proprietary data model and billing logic to ensure that the metrics they collect are formatted correctly and don’t cause a “cardinality explosion ” that spikes your monthly bill.
• Kubernetes (GKE) & Helm: Since the collector typically runs as a DaemonSet or Sidecar in GCP, mastery here ensures the monitoring pipeline is resilient, scales with your clusters, and is easily updated via standardized charts.
• Terraform / Terragrunt: High-quality observability must be “codified ” rather than manual; using IaC ensures that every new GCP project or resource is automatically onboarded with the correct labels and alert policies from day one.
• PromQL & MQL (Query Languages): Collecting the data is only half the battle; the candidate needs these languages to build the complex dashboards and SLO (Service Level Objective) ratios that actually prove the “value-add ” to your engineering teams.