Splunk Observability Engineer — AWS & OpenTelemetry

System One; Birmingham (AL); On-site; USD 90,000 - 150,000; Full time

Job summary

System One is seeking a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting. This hands-on role focuses on running, upgrading and supporting Splunk Enterprise on EC2 in a secure, scalable environment.

You will manage data ingestion, indexing, search, dashboards and alerts, write complex SPL queries, build monitoring dashboards and participate in on-call rotation to ensure incident response.

Qualifications

• Strong Splunk admin experience in large production setups.
• Advanced SPL skills.
• Strong AWS experience: EC2, ALB, CloudWatch.
• Hands on OpenTelemetry experience, including Splunk integration.
• Python for automation.
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON.
• Strong troubleshooting and clear communication skills.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Fix ingestion failures, missing data, volume spikes and outages.
• Restore and replay data after interruptions.
• Reduce noise, storage use and cost.
• Connect Splunk with CloudWatch and OpenTelemetry.
• Build and maintain OpenTelemetry agent and collector packages.
• Automate tasks with Python.
• Handle patching and security fixes.
• Lead troubleshooting calls across teams.
• Take part in on call rotation.

Skills

Splunk admin

SPL queries

AWS

OpenTelemetry

Python automation

Troubleshooting

Communication

CI/CD

Terraform IaC

JSON

Tools

GitLab

GitHub

Terraform Enterprise

OpenTelemetry

Job description

System One is seeking a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting. This hands-on role focuses on running, upgrading and supporting Splunk Enterprise on EC2 in a secure, scalable environment.

You will manage data ingestion, indexing, search, dashboards and alerts, write complex SPL queries, build monitoring dashboards and participate in on-call rotation to ensure incident response.