Splunk Observability Engineer

System One; Lafayette (LA); On-site; USD 100,000 - 130,000; Full time

Job summary

System One is seeking a hands-on Splunk Observability Engineer to support large Splunk platforms in an AWS production setting. The role covers running, upgrading, and supporting Splunk Enterprise on EC2, managing ingestion, indexing, search, dashboards, and alerts, and reducing cost and noise.

You will write SPL queries, build dashboards, fix data issues, and automate tasks with Python while coordinating across teams.

Qualifications

• Experience administering Splunk in large production deployments (forwarders, indexers, search heads).
• Strong SPL query skills and data lifecycle management experience.
• Hands-on AWS experience (EC2, CloudWatch).
• Experience with OpenTelemetry integration with Splunk.
• Automation with Python for operational tasks.
• Familiarity with GitLab, GitHub, Terraform Enterprise and CI/CD processes.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Fix ingestion failures, data gaps, outages and spikes.
• Restore and replay data after interruptions.
• Reduce noise, storage use and cost in Splunk deployments.
• Integrate Splunk with CloudWatch and OpenTelemetry.
• Build and maintain OpenTelemetry agent and collector packages.
• Automate tasks with Python.
• Handle patching and security fixes.
• Lead troubleshooting calls across teams and participate in on-call rotation.

Skills

Splunk admin

SPL queries

AWS expertise

OpenTelemetry

Python automation

CI/CD

JSON

Troubleshooting

Tools

Splunk Enterprise

AWS

OpenTelemetry

Python

GitLab

GitHub

Terraform Enterprise

CI/CD tooling

JSON

Job description

Splunk Observability Engineer

Hands on role supporting large Splunk and observability platforms in an AWS production setting.

• Run, upgrade and support Splunk Enterprise on AWS (EC2)
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery
• Write and tune complex SPL queries
• Build dashboards for monitoring and incident triage
• Fix ingestion failures, missing data, volume spikes and outages
• Restore and replay data after interruptions
• Reduce noise, storage use and cost
• Connect Splunk with CloudWatch and OpenTelemetry
• Build and maintain OpenTelemetry agent and collector packages
• Automate tasks with Python
• Handle patching and security fixes
• Lead troubleshooting calls across teams
• Take part in on call rotation

Locations & Eligibility

• no C2C with 3rd parties, no contractors - W2 salaried candidates only
• Applicants must be U.S. citizens, Green Card holders or current H-1B visa holders willing to transfer their visas ( H-1B visa transfer )
• Must be able to work on site in either of the following locations : Lafayette, LA - Knoxville, TN - Columbia, SC - Birmingham, AL

Requirements

• Strong Splunk admin experience in large production setups (forwarders, indexers, search heads)
• Advanced SPL skills
• Strong AWS experience: EC2, ALB, CloudWatch
• Hands on OpenTelemetry experience, including Splunk integration
• Python for automation
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON
• Strong troubleshooting and clear communication skills

Ref: #404-IT Pittsburgh