Splunk Observability Engineer

System One; Columbia (SC); On-site; USD 120,000 - 150,000; Full time

Job summary

System One seeks a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting, on-site in multiple U.S. locations.

You will run, upgrade, and support Splunk Enterprise on AWS (EC2) and manage the full data lifecycle from ingestion to recovery. You will write and tune SPL queries, build dashboards for monitoring and incident triage, fix ingestion issues, automate tasks with Python, and participate in on-call rotation while coordinating across

Qualifications

• Strong Splunk admin experience in large production setups.
• Advanced SPL skills.
• Hands-on AWS experience (EC2, CloudWatch).
• OpenTelemetry integration with Splunk.
• Python for automation.
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON.
• Strong troubleshooting and communication skills.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Fix ingestion failures, missing data, volume spikes and outages.
• Restore and replay data after interruptions.
• Reduce noise, storage use and cost.
• Connect Splunk with CloudWatch and OpenTelemetry.
• Build and maintain OpenTelemetry agent and collector packages.
• Automate tasks with Python.
• Handle patching and security fixes.
• Lead troubleshooting calls across teams.
• Take part in on call rotation.

Skills

Splunk admin

Advanced SPL

Python

OpenTelemetry

AWS CloudWatch

CI/CD

Tools

GitHub/GitLab

Terraform Enterprise

JSON

AWS (EC2/ALB)

Job description

Splunk Observability Engineer

Hands on role supporting large Splunk and observability platforms in an AWS production setting.

• Run, upgrade and support Splunk Enterprise on AWS (EC2)
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery
• Write and tune complex SPL queries
• Build dashboards for monitoring and incident triage
• Fix ingestion failures, missing data, volume spikes and outages
• Restore and replay data after interruptions
• Reduce noise, storage use and cost
• Connect Splunk with CloudWatch and OpenTelemetry
• Build and maintain OpenTelemetry agent and collector packages
• Automate tasks with Python
• Handle patching and security fixes
• Lead troubleshooting calls across teams
• Take part in on call rotation

Locations & Eligibility

• no C2C with 3rd parties, no contractors - W2 salaried candidates only
• Applicants must be U.S. citizens, Green Card holders or current H-1B visa holders willing to transfer their visas ( H-1B visa transfer )
• Must be able to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL

Requirements

• Strong Splunk admin experience in large production setups (forwarders, indexers, search heads)
• Advanced SPL skills
• Strong AWS experience: EC2, ALB, CloudWatch
• Hands on OpenTelemetry experience, including Splunk integration
• Python for automation
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON
• Strong troubleshooting and clear communication skills

Ref: #404-IT Pittsburgh