Splunk Observability Engineer

MethodHub; United States; On-site; USD 120,000 - 180,000; Full time

Job summary

MethodHub is seeking an experienced full-stack observability engineer to design and implement a complete visibility strategy using the Splunk Observability Cloud and Splunk Enterprise. You will drive proactive pattern-based incident prevention and enable engineers with 360-degree system health visibility.

The role requires deep expertise in telemetry, data aggregation, anomaly detection, and dashboarding to correlate logs, traces, and metrics across services.

Qualifications

• Experience designing observability pipelines across metrics, logs and traces.
• Proficient in SPL, SignalFlow, and distributed tracing instrumentation.
• Familiar with data scrubbing and edge processing to reduce license costs.
• Ability to build dashboards with contextual drill-downs (Glass Tables / ITSI).

Responsibilities

• Architect ingestion of Metrics, Logs and Traces using OpenTelemetry collectors.
• Develop aggregation logic to reduce noise while preserving signal for troubleshooting.
• Use SPL and SignalFlow for pattern analysis and anomaly detection.
• Build executive and technical dashboards linking logs, traces, and metrics.

Skills

Logging-in-Context

SignalFlow

OpenTelemetry

Distributed Tracing

Anomaly Detection

Data Scrubbing

Pattern Discovery

Dashboards

Glass Tables

Tools

Splunk Ingest Actions

Edge Processors

Metric Finder

MDetector

OTel Collector

Job description

To design, implement, and optimize a full-stack observability strategy using the Splunk Observability Cloud (formerly SignalFx) and Splunk Enterprise/Cloud. You will ensure that engineering teams have 360-degree visibility into system health, moving the organization from reactive "firefighting" to proactive "pattern-based" incident prevention.

Key Responsibilities:

• Data Orchestration: Architect the ingestion of the "Three Pillars" (Metrics, Logs, Traces) using OpenTelemetry (OTel) collectors.
• Aggregation Strategy: Develop logic to aggregate high-cardinality data to reduce "noise" while maintaining "signal" for troubleshooting.
• Analytical Modeling: Use SPL (Search Processing Language) and SignalFlow to perform pattern analysis, detecting anomalies before they trigger traditional threshold alerts.
• Visual Storytelling: Build executive and technical dashboards that correlate disparate data points (e.g., showing how a spike in 500-errors in Logs relates to a specific span in a Trace).

Required Hands on Technical Skills:

1. Telemetry & Data Specialization

• Logs: Proficiency in "Logging-in-Context." You must be able to link logs directly to trace IDs so developers can jump from a failing trace to the specific line of code in the logs.
• Metrics: Expertise in SignalFlow (Splunk’s background streaming analytics language). You should know how to calculate percentiles ($P95, $P99$), rates of change, and historical averages.
• Traces: Deep understanding of Distributed Tracing. You must know how to instrument applications (Java, Python, Go) to capture spans and identify bottlenecks in microservices.

2. Pattern Analysis & Aggregation

• Anomaly Detection: Ability to configure Metric Finder and MDetector using standard deviations or "Mean Absolute Deviation" to find outliers.
• Data Scrubbing: Skills in using Splunk Ingest Actions or Edge Processors to filter, mask, or aggregate data at the edge to save on license costs and improve search speed.
• Pattern Discovery: Using Splunk’s machine learning commands (e.g., findkeywords, cluster) to group millions of log events into a few dozen "patterns" for faster root cause analysis.

3. Hands on - Dashboards & Visualization

• High-Cardinality Handling: Designing dashboards that don’t "break" when viewing thousands of containers.
• Contextual Drill-downs: Building "Glass Tables" (in ITSI) or Unified Dashboards that allow a user to click a metric and immediately see the associated logs.
• Frameworks: Familiarity with the Dashboard Studio and JSON-based dashboard definitions for version control (GitOps).

Preferred Qualifications & Certifications:

• Splunk Cloud Certified Metrics User: Focuses on the metrics and alerting side.
• Splunk Core Certified Power User: Essential for mastering complex SPL for log analysis.
• OpenTelemetry Expert: Knowledge of the OTel Collector configuration (receivers, processors, exporters) is currently the most "in-demand" skill for this role.