Senior Splunk & Observability Engineer — Hybrid & AWS

System One, Inc.; Lafayette (LA); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health and welfare benefits

401(k) plan

Job summary

System One, Inc. is seeking a Senior Splunk & Observability Engineer to join a hybrid team in Lafayette, LA. You will manage Splunk on AWS, develop SPL queries, and build dashboards for production monitoring and incident triage.

The role requires 5+ years in Splunk, strong AWS experience, and proficiency in Python automation. You will participate in on-call rotations and coordinate cross-team incident response.

Qualifications

• 5+ years hands-on Splunk Enterprise administration in large production environments.
• Strong Splunk ecosystem knowledge: ingestion, forwarders, indexers, search heads, dashboards, alerts, recovery.
• Advanced SPL query development and performance optimization.
• Experience deploying Splunk on AWS (EC2, ALB, CloudWatch).
• Experience building complex Splunk dashboards for monitoring and incident use cases.
• Knowledge of Open Telemetry, including agents, collectors, instrumentation, and Splunk integration.
• Experience with Python for automation.
• Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, JSON.
• Strong troubleshooting for ingestion failures, data volume, outages, and recovery.
• Bachelor's degree in computer science, information systems, or related field.

Responsibilities

• Support enterprise-scale observability platforms in an AWS-based production environment.
• Manage Splunk Enterprise on AWS: ingestion, indexing, dashboards, alerts, and recovery.
• Develop and optimize SPL queries for performance and outages.
• Build enterprise dashboards for production monitoring and incident triage.
• Integrate with AWS, CloudWatch, and Open Telemetry; develop Python automation.
• Assist with upgrades, patching, and vulnerability mitigation.
• Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD for deployment.
• Lead troubleshooting calls and coordinate incident response across teams.
• Participate in on-call rotation after hours.

Skills

Troubleshooting

Cross-team coordination

Python automation

Incident response

Open Telemetry knowledge

Education

Bachelor's degree in CS/IS

Tools

Splunk Enterprise

AWS

CloudWatch

GitLab

Terraform Enterprise

GitHub

Open Telemetry

Python

JSON

Job description

System One, Inc. is seeking a Senior Splunk & Observability Engineer to join a hybrid team in Lafayette, LA. You will manage Splunk on AWS, develop SPL queries, and build dashboards for production monitoring and incident triage.

The role requires 5+ years in Splunk, strong AWS experience, and proficiency in Python automation. You will participate in on-call rotations and coordinate cross-team incident response.