Senior Splunk & Observability Engineer (AWS)

System One; Birmingham (AL); On-site; USD 120,000 - 180,000; Full time

Job summary

System One seeks a Senior Splunk & Observability Engineer to manage enterprise-scale observability on AWS. You will handle Splunk Enterprise, data lifecycle, and dashboards, while integrating OpenTelemetry and Python automation across multiple stacks.

The role demands hands-on expertise, strong troubleshooting, and the ability to coordinate across cloud, app, security, and prod-support teams. On-call rotation may apply.

Qualifications

• 5+ years of hands on Splunk Enterprise administration and engineering in large production environments.
• Strong understanding of the Splunk ecosystem: ingestion, forwarders, indexers, search heads, dashboards, alerts, recovery.

Responsibilities

• Administer, engineer, upgrade, and support Splunk Enterprise in AWS.
• Deploy, configure, and troubleshoot Splunk components on EC2.
• Manage end-to-end Splunk data lifecycle: ingestion, indexing, search, dashboards, alerts, retention, recovery.
• Develop and optimize SPL queries and troubleshoot performance issues.
• Build enterprise dashboards for production monitoring, resiliency, incident response, outage triage.
• Lead on-call rotations and coordinate incident investigations across teams.

Skills

Splunk Enterprise

SPL queries

Dashboards & alerts

Production troubleshooting

On-call coordination

Cross-team collaboration

Python automation

Education

Bachelor's degree in Computer Science

Tools

AWS

Amazon EC2

OpenTelemetry

Terraform Enterprise

GitLab

GitHub

JSON

Job description

System One seeks a Senior Splunk & Observability Engineer to manage enterprise-scale observability on AWS. You will handle Splunk Enterprise, data lifecycle, and dashboards, while integrating OpenTelemetry and Python automation across multiple stacks.

The role demands hands-on expertise, strong troubleshooting, and the ability to coordinate across cloud, app, security, and prod-support teams. On-call rotation may apply.