Splunk Observability Engineer

System One; Knoxville (TN); On-site; USD 110,000 - 140,000; Full time

Job summary

System One is seeking a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting. This on-site role covers administration, data lifecycle, and incident triage across multiple locations.

You will upgrade and operate Splunk on EC2, develop SPL queries, build dashboards, automate with Python, and collaborate during on-call rotations to reduce noise and outages.

Qualifications

• Strong Splunk admin experience in large production deployments.
• Proficient in building dashboards, alerts, and data lifecycle management.
• Hands-on AWS (EC2, ALB, CloudWatch) and OpenTelemetry integration.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage data ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Lead troubleshooting calls across teams and participate in on-call rotation.

Skills

Splunk admin experience

Advanced SPL

AWS experience

OpenTelemetry experience

Python for automation

Troubleshooting

Communication skills

Tools

GitLab

GitHub

Terraform Enterprise

CI/CD

JSON

Job description

Splunk Observability Engineer

Hands on role supporting large Splunk and observability platforms in an AWS production setting.

• Run, upgrade and support Splunk Enterprise on AWS (EC2)
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery
• Write and tune complex SPL queries
• Build dashboards for monitoring and incident triage
• Fix ingestion failures, missing data, volume spikes and outages
• Restore and replay data after interruptions
• Reduce noise, storage use and cost
• Connect Splunk with CloudWatch and OpenTelemetry
• Build and maintain OpenTelemetry agent and collector packages
• Automate tasks with Python
• Handle patching and security fixes
• Lead troubleshooting calls across teams
• Take part in on call rotation

Locations & Eligibility

• no C2C with 3rd parties, no contractors - W2 salaried candidates only
• Applicants must be U.S. citizens, Green Card holders or current H-1B visa holders willing to transfer their visas ( H-1B visa transfer )
• Must be able to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL

Requirements

• Strong Splunk admin experience in large production setups (forwarders, indexers, search heads)
• Advanced SPL skills
• Strong AWS experience: EC2, ALB, CloudWatch
• Hands on OpenTelemetry experience, including Splunk integration
• Python for automation
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON
• Strong troubleshooting and clear communication skills

Ref: #404-IT Pittsburgh