Splunk Observability Engineer — AWS & OpenTelemetry

System One; Knoxville (TN); On-site; USD 110,000 - 140,000; Full time

Job summary

System One is seeking a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting. This on-site role covers administration, data lifecycle, and incident triage across multiple locations.

You will upgrade and operate Splunk on EC2, develop SPL queries, build dashboards, automate with Python, and collaborate during on-call rotations to reduce noise and outages.

Qualifications

• Strong Splunk admin experience in large production deployments.
• Proficient in building dashboards, alerts, and data lifecycle management.
• Hands-on AWS (EC2, ALB, CloudWatch) and OpenTelemetry integration.

Responsibilities

• Run, upgrade and support Splunk Enterprise on AWS (EC2).
• Manage data ingestion, indexing, search, dashboards, alerts, retention and recovery.
• Write and tune complex SPL queries.
• Build dashboards for monitoring and incident triage.
• Lead troubleshooting calls across teams and participate in on-call rotation.

Skills

Splunk admin experience

Advanced SPL

AWS experience

OpenTelemetry experience

Python for automation

Troubleshooting

Communication skills

Tools

GitLab

GitHub

Terraform Enterprise

CI/CD

JSON

Job description

System One is seeking a Splunk Observability Engineer to support large Splunk and observability platforms in an AWS production setting. This on-site role covers administration, data lifecycle, and incident triage across multiple locations.

You will upgrade and operate Splunk on EC2, develop SPL queries, build dashboards, automate with Python, and collaborate during on-call rotations to reduce noise and outages.