Senior Splunk & Observability Engineer - AWS

System One, Inc.; Columbia (SC); On-site; USD 120,000 - 150,000; Full time

Job summary

System One, Inc. seeks a Senior Splunk & Observability Engineer to manage enterprise-scale observability platforms in an AWS-based production environment. This senior role combines Splunk administration, AWS support, observability, and automation.

The engineer will manage Splunk Enterprise on AWS, handling ingestion, indexing, dashboards, alerts, retention, recovery, and performance optimization. Collaboration with AWS, CloudWatch, OpenTelemetry, and Python automation is essential to improve

Qualifications

• 5+ years of hands on Splunk Enterprise administration in large production environments.
• Strong AWS production experience with EC2, Application Load Balancer, and CloudWatch.
• Advanced SPL query development and performance optimization.
• Experience building or maintaining OpenTelemetry agent and collector packages.
• Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON.

Responsibilities

• Administer, engineer, upgrade, and support Splunk Enterprise in AWS.
• Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2.
• Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery.
• Develop and optimize complex SPL queries and troubleshoot query performance issues.
• Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
• Troubleshoot ingestion failures, problematic indexes, missing or delayed data, data volume growth, and platform availability issues.
• Restore data flow and replay data following ingestion or platform interruptions.
• Integrate Splunk, CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts.
• Develop Python based automation for observability and platform operations.
• Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms.
• Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation.
• Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and deployments.
• Lead technical troubleshooting calls and coordinate incident investigation across teams.
• Participate in an on call rotation and provide after hours support when required.

Skills

Splunk Enterprise

SPL query development

AWS (EC2)

OpenTelemetry

Python automation

Education

Bachelor's degree in Computer Science

Tools

GitLab

GitHub

Terraform Enterprise

JSON

Job description

System One, Inc. seeks a Senior Splunk & Observability Engineer to manage enterprise-scale observability platforms in an AWS-based production environment. This senior role combines Splunk administration, AWS support, observability, and automation.

The engineer will manage Splunk Enterprise on AWS, handling ingestion, indexing, dashboards, alerts, retention, recovery, and performance optimization. Collaboration with AWS, CloudWatch, OpenTelemetry, and Python automation is essential to improve