Splunk Observability Engineer — AWS & OpenTelemetry

System One; Columbia (SC); On-site; USD 120,000 - 150,000; Full time

Job summary

System One seeks a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting, on-site in multiple U.S. locations.

You will run, upgrade, and support Splunk Enterprise on AWS (EC2) and manage the full data lifecycle from ingestion to recovery. You will write and tune SPL queries, build dashboards for monitoring and incident triage, fix ingestion issues, automate tasks with Python, and participate in on-call rotation while coordinating across

Qualifications

• Strong Splunk admin experience in large production setups.
• Advanced SPL skills.
• Hands-on AWS experience (EC2, CloudWatch).
• OpenTelemetry integration with Splunk.
• Python for automation.
• GitLab, GitHub, Terraform Enterprise, CI/CD, JSON.
• Strong troubleshooting and communication skills.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage the full data lifecycle: ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Fix ingestion failures, missing data, volume spikes and outages.
• Restore and replay data after interruptions.
• Reduce noise, storage use and cost.
• Connect Splunk with CloudWatch and OpenTelemetry.
• Build and maintain OpenTelemetry agent and collector packages.
• Automate tasks with Python.
• Handle patching and security fixes.
• Lead troubleshooting calls across teams.
• Take part in on call rotation.

Skills

Splunk admin

Advanced SPL

Python

OpenTelemetry

AWS CloudWatch

CI/CD

Tools

GitHub/GitLab

Terraform Enterprise

JSON

AWS (EC2/ALB)

Job description

System One seeks a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting, on-site in multiple U.S. locations.

You will run, upgrade, and support Splunk Enterprise on AWS (EC2) and manage the full data lifecycle from ingestion to recovery. You will write and tune SPL queries, build dashboards for monitoring and incident triage, fix ingestion issues, automate tasks with Python, and participate in on-call rotation while coordinating across