Senior Splunk & Observability Engineer

System One, Inc.; Lafayette (LA); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health and welfare benefits

401(k) plan

Job summary

System One, Inc. is seeking a Senior Splunk & Observability Engineer to join a hybrid team in Lafayette, LA. You will manage Splunk on AWS, develop SPL queries, and build dashboards for production monitoring and incident triage.

The role requires 5+ years in Splunk, strong AWS experience, and proficiency in Python automation. You will participate in on-call rotations and coordinate cross-team incident response.

Qualifications

• 5+ years hands-on Splunk Enterprise administration in large production environments.
• Strong Splunk ecosystem knowledge: ingestion, forwarders, indexers, search heads, dashboards, alerts, recovery.
• Advanced SPL query development and performance optimization.
• Experience deploying Splunk on AWS (EC2, ALB, CloudWatch).
• Experience building complex Splunk dashboards for monitoring and incident use cases.
• Knowledge of Open Telemetry, including agents, collectors, instrumentation, and Splunk integration.
• Experience with Python for automation.
• Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, JSON.
• Strong troubleshooting for ingestion failures, data volume, outages, and recovery.
• Bachelor's degree in computer science, information systems, or related field.

Responsibilities

• Support enterprise-scale observability platforms in an AWS-based production environment.
• Manage Splunk Enterprise on AWS: ingestion, indexing, dashboards, alerts, and recovery.
• Develop and optimize SPL queries for performance and outages.
• Build enterprise dashboards for production monitoring and incident triage.
• Integrate with AWS, CloudWatch, and Open Telemetry; develop Python automation.
• Assist with upgrades, patching, and vulnerability mitigation.
• Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD for deployment.
• Lead troubleshooting calls and coordinate incident response across teams.
• Participate in on-call rotation after hours.

Skills

Troubleshooting

Cross-team coordination

Python automation

Incident response

Open Telemetry knowledge

Education

Bachelor's degree in CS/IS

Tools

Splunk Enterprise

AWS

CloudWatch

GitLab

Terraform Enterprise

GitHub

Open Telemetry

Python

JSON

Job description

Job Title: Senior Splunk & Observability Engineer
Location: Lafayette, LA
Type: Permanent Full Time
Work Model: Hybrid – onsite and remote

Responsibilities

• Support enterprise scale observability platforms within a complex, AWS-based production environment.
• Manage and support Splunk Enterprise running on AWS, covering the full data lifecycle including source onboarding, ingestion, indexing, search, dashboards, alerts, retention, and recovery.
• Develop and optimize SPL queries to troubleshoot ingestion and performance issues, enhance platform stability, support upgrades, vulnerability remediation, and data recovery after service interruptions.
• Build and maintain enterprise dashboards for production monitoring, resiliency, incident response, and outage triage.
• Integrate with AWS, Amazon CloudWatch, and Open Telemetry; develop Python-based automation for observability and platform operations.
• Support platform upgrades, patching, configuration changes, and vulnerability mitigation.
• Use GitLab, GitHub, Terraform Enterprise, JSON, and CI/CD tools for infrastructure and automation deployment.
• Lead technical troubleshooting calls and coordinate incident investigation and remediation across multiple teams.
• Participate in an on‑call rotation for after‑hours support related to production incidents, upgrades, and planned changes.

Requirements

• 5+ years of hands‑on Splunk Enterprise administration and engineering experience in large production environments.
• Strong understanding of the Splunk ecosystem, including data ingestion, forwarders, indexers, search heads, dashboards, alerts, and recovery.
• Advanced SPL query development and performance optimization skills.
• Hands‑on experience deploying, configuring, upgrading, troubleshooting, and supporting Splunk on AWS.
• Strong AWS production experience, particularly with Amazon EC2, Application Load Balancer, and Amazon CloudWatch.
• Experience building complex Splunk dashboards for production monitoring, resiliency, incident triage, and operational use cases.
• Strong understanding of Open Telemetry, including agents, collectors, logs, metrics, instrumentation, and Splunk integration.
• Experience building or maintaining Open Telemetry agent and collector packages.
• Working knowledge of Python for automation and operational tooling.
• Experience with GitLab, Terraform Enterprise, CI/CD, GitHub, and JSON.
• Strong troubleshooting skills including ingestion failures, data volume issues, platform outages, and data recovery.
• Experience with platform upgrades, patching, vulnerability remediation, and controlled production changes.
• Ability to lead technical troubleshooting calls, communicate clearly, and coordinate across cloud, application, security, database, and support teams.
• Educational Requirements: Bachelor's degree in computer science, Information Systems, or a related field.

System One, and its subsidiaries including Mountain Ltd., are leaders in delivering outsourced services and workforce solutions across North America. We help clients get work done more efficiently and economically, without compromising quality. System One not only serves as a valued partner for our clients, but we offer eligible employees health and welfare benefits coverage options including medical, dental, vision, spending accounts, life insurance, voluntary plans, as well as participation in a 401(k) plan.

System One is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity, age, national origin, disability, family care or medical leave status, genetic information, veteran status, marital status, or any other characteristic protected by applicable federal, state, or local law.