Senior Splunk & Observability Engineer (AWS, OpenTelemetry)

Energy Jobline ZR; Knoxville (TN); On-site; USD 140,000 - 190,000; Full time

Job summary

Energy Jobline ZR seeks an experienced Senior Splunk & Observability Engineer to manage enterprise observability platforms in a mixed AWS production environment. The role combines Splunk administration with AWS operations, OpenTelemetry, and Python automation to improve monitoring and resiliency.

The candidate will develop SPL queries, build dashboards for production stability, and lead incident investigations across teams. A strong background in Splunk, AWS, and automation is essential.

Qualifications

• 5+ years of hands on Splunk Enterprise administration and engineering experience in large production environments.
• Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, indexes, dashboards, alerts, and recovery.
• Advanced SPL query development and performance optimization skills.
• Hands on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS.
• Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch.

Responsibilities

• Administer, engineer, upgrade, and support Splunk Enterprise in AWS.
• Deploy, configure, and troubleshoot Splunk components hosted on Amazon EC2.
• Manage the end to end Splunk data lifecycle, including ingestion, indexing, search, dashboards, alerts, retention, and recovery.
• Develop and optimize complex SPL queries and troubleshoot query performance issues.
• Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
• Troubleshoot ingestion failures, problematic indexes, missing or delayed data, unexpected data volume growth, and platform availability issues.
• Restore data flow and replay data following ingestion or platform interruptions.
• Optimize data ingestion and retention to reduce noise, storage consumption, and operating costs.
• Integrate Splunk, Amazon CloudWatch, and OpenTelemetry for logs, metrics, dashboards, and alerts.
• Develop Python based automation for observability and platform operations.
• Build and maintain OpenTelemetry agent and collector packages across multiple technology platforms.
• Support Splunk upgrades, patching, configuration changes, plugins, and vulnerability remediation.
• Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tooling to support infrastructure, configuration, automation, and package deployments.
• Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams.
• Participate in an on call rotation and provide after hours support when required for production incidents, upgrades, and planned changes.

Skills

Splunk Admin

AWS EC2

OpenTelemetry

Python automation

SPL queries

Dashboards

CI/CD tooling

Education

Bachelor's degree

Tools

Terraform Enterprise

GitLab

GitHub

JSON

Job description

Energy Jobline ZR seeks an experienced Senior Splunk & Observability Engineer to manage enterprise observability platforms in a mixed AWS production environment. The role combines Splunk administration with AWS operations, OpenTelemetry, and Python automation to improve monitoring and resiliency.

The candidate will develop SPL queries, build dashboards for production stability, and lead incident investigations across teams. A strong background in Splunk, AWS, and automation is essential.