Splunk Observability Engineer

System One; Birmingham (AL); On-site; USD 90,000 - 150,000; Full time

Job summary

System One is seeking a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting. This hands-on role focuses on running, upgrading and supporting Splunk Enterprise on EC2 in a secure, scalable environment.

You will manage data ingestion, indexing, search, dashboards and alerts, write complex SPL queries, build monitoring dashboards and participate in on-call rotation to ensure incident response.

Qualifications

• Strong Splunk admin experience in large production setups.
• Advanced SPL skills.
• Strong AWS experience: EC2, ALB, CloudWatch.
• Hands on OpenTelemetry experience, including Splunk integration.
• Python for automation.
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON.
• Strong troubleshooting and clear communication skills.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Fix ingestion failures, missing data, volume spikes and outages.
• Restore and replay data after interruptions.
• Reduce noise, storage use and cost.
• Connect Splunk with CloudWatch and OpenTelemetry.
• Build and maintain OpenTelemetry agent and collector packages.
• Automate tasks with Python.
• Handle patching and security fixes.
• Lead troubleshooting calls across teams.
• Take part in on call rotation.

Skills

Splunk admin

SPL queries

AWS

OpenTelemetry

Python automation

Troubleshooting

Communication

CI/CD

Terraform IaC

JSON

Tools

GitLab

GitHub

Terraform Enterprise

OpenTelemetry

Job description

Splunk Observability Engineer

Hands on role supporting large Splunk and observability platforms in an AWS production setting.

• Run, upgrade and support Splunk Enterprise on AWS (EC2)
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery
• Write and tune complex SPL queries
• Build dashboards for monitoring and incident triage
• Fix ingestion failures, missing data, volume spikes and outages
• Restore and replay data after interruptions
• Reduce noise, storage use and cost
• Connect Splunk with CloudWatch and OpenTelemetry
• Build and maintain OpenTelemetry agent and collector packages
• Automate tasks with Python
• Handle patching and security fixes
• Lead troubleshooting calls across teams
• Take part in on call rotation

Work Conditions

• No C2C with third parties, no contractors - W2 salaried candidates only
• Applicants must be U.S. citizens, Green Card holders or current H-1B visa holders willing to transfer their visas ( H-1B visa transfer )
• Must be able to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL

Requirements

• Strong Splunk admin experience in large production setups (forwarders, indexers, search heads)
• Advanced SPL skills
• Strong AWS experience: EC2, ALB, CloudWatch
• Hands on OpenTelemetry experience, including Splunk integration
• Python for automation
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON
• Strong troubleshooting and clear communication skills