Observability Engineer

STRAVEXA

United States

On-site

USD 120,000 - 180,000

Full time

25 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

STAFFXPERT is seeking a Splunk Observability Engineer to architect and optimize a full-stack observability strategy using Splunk Observability Cloud and Splunk Enterprise. You will ensure 360-degree visibility into system health and drive proactive incident prevention through advanced data modeling and dashboards.

The role focuses on ingestion of Metrics, Logs, and Traces, with hands-on work on OpenTelemetry, SPL, and SignalFlow to detect anomalies and present insights through Glass Tables or

Qualifications

  • Experience with telemetry data ingestion and OpenTelemetry collectors.
  • Proficiency in Splunk SPL, SignalFlow, and anomaly detection.
  • Experience instrumenting applications for distributed tracing and building dashboards.

Responsibilities

  • Data Orchestration: Architect the ingestion of Metrics, Logs, and Traces using OpenTelemetry collectors.
  • Aggregation Strategy: Develop logic to reduce noise while preserving signal for troubleshooting.
  • Analytical Modeling: Use SPL and SignalFlow to perform pattern analysis and detect anomalies.
  • Visual Storytelling: Build dashboards that correlate data points across logs and traces.

Skills

Logging-in-Context
SignalFlow
Distributed Tracing
Anomaly Detection
Data Scrubbing
Pattern Discovery
Dashboard Studio
Glass Tables
JSON Dashboards

Tools

OTel Collector
Splunk Ingest Actions
Edge Processors
SPL
SignalFlow
ITSI

Job description

Position – Splunk Observability Engineer
Position Summary

STAFFXPERT is looking for a SplunK implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive "firefighting" to proactive "pattern-based" incident prevention.

Key Responsibilities
  • Data Orchestration: Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) using OpenTelemetry (OTel) collectors.
  • Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.
  • Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.
  • Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).
Required Hands on Technical Skills
1. Telemetry & Data Specialization
  • Logs: Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.
  • Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, P99$), rates of change, and historical averages.
  • Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.
2. Pattern Analysis & Aggregation
  • Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.
  • Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.
  • Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen "patterns" for faster root cause analysis.
3. Hands on - Dashboards & Visualization
  • High-Cardinality Handling: Designing dashboards that don’t "break" when viewing thousands of containers.
  • Contextual Drill-downs: Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.
  • Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).
Preferred Qualifications & Certifications
  • Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.
  • Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.
  • OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most "in-demand" skill for this role
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Observability Engineer (Splunk / SignalFx) - US Citizen or Green Card only
Senior Observability Engineer (Splunk / SignalFx) - US Citizen or Green Card only

Nextiya LLC • Fairfax (VA)

On-site
USD 120,000 - 160,000
Senior Splunk Observability Engineer
Senior Splunk Observability Engineer

STRAVEXA • United States

On-site
USD 120,000 - 180,000
Splunk Obervability Cloud
Splunk Obervability Cloud

METRIX IT SOLUTIONS INC • Charlotte (NC)

On-site
USD 100,000 - 130,000
Splunk Subject Matter Expert (SME) & Enterprise Monitoring Engineer
Splunk Subject Matter Expert (SME) & Enterprise Monitoring Engineer

Empower Professionals Inc - Talent & IT Services • Frisco (TX)

Hybrid
USD 120,000 - 150,000
Senior Splunk & Observability Engineer
Senior Splunk & Observability Engineer

System One, Inc. • Knoxville (TN)

On-site
USD 120,000 - 180,000
Senior Splunk & Observability Engineer
Senior Splunk & Observability Engineer

System One • Lafayette (LA)

On-site
USD 120,000 - 180,000
Senior Splunk & Observability Engineer
Senior Splunk & Observability Engineer

Okta Inc. • New York (NY)

On-site
USD 174,000 - 238,000
Senior Splunk Administrator
Senior Splunk Administrator

PRI Technology • Holmdel Township (NJ)

On-site
USD 100,000 - 130,000
Senior/Lead Site Reliability Engineer Observability
Senior/Lead Site Reliability Engineer Observability

Tata Consultancy Services Limited • San Jose (CA)

On-site
USD 94,000 - 130,000
Lead Site Reliability Engineer – Observability
Lead Site Reliability Engineer – Observability

Rishi Writes • United States

Remote
USD 140,000 - 180,000