Splunk Observability Engineer — AWS & OpenTelemetry

System One; Lafayette (LA); On-site; USD 100,000 - 130,000; Full time

Job summary

System One is seeking a hands-on Splunk Observability Engineer to support large Splunk platforms in an AWS production setting. The role covers running, upgrading, and supporting Splunk Enterprise on EC2, managing ingestion, indexing, search, dashboards, and alerts, and reducing cost and noise.

You will write SPL queries, build dashboards, fix data issues, and automate tasks with Python while coordinating across teams.

Qualifications

• Experience administering Splunk in large production deployments (forwarders, indexers, search heads).
• Strong SPL query skills and data lifecycle management experience.
• Hands-on AWS experience (EC2, CloudWatch).
• Experience with OpenTelemetry integration with Splunk.
• Automation with Python for operational tasks.
• Familiarity with GitLab, GitHub, Terraform Enterprise and CI/CD processes.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Fix ingestion failures, data gaps, outages and spikes.
• Restore and replay data after interruptions.
• Reduce noise, storage use and cost in Splunk deployments.
• Integrate Splunk with CloudWatch and OpenTelemetry.
• Build and maintain OpenTelemetry agent and collector packages.
• Automate tasks with Python.
• Handle patching and security fixes.
• Lead troubleshooting calls across teams and participate in on-call rotation.

Skills

Splunk admin

SPL queries

AWS expertise

OpenTelemetry

Python automation

CI/CD

JSON

Troubleshooting

Tools

Splunk Enterprise

AWS

OpenTelemetry

Python

GitLab

GitHub

Terraform Enterprise

CI/CD tooling

JSON

Job description

System One is seeking a hands-on Splunk Observability Engineer to support large Splunk platforms in an AWS production setting. The role covers running, upgrading, and supporting Splunk Enterprise on EC2, managing ingestion, indexing, search, dashboards, and alerts, and reducing cost and noise.

You will write SPL queries, build dashboards, fix data issues, and automate tasks with Python while coordinating across teams.