Senior Splunk Engineer - Infrastructure Operations

GovCIO; United States; On-site; USD 105,000 - 145,000; Full time

Job summary

GovCIO is hiring a Senior Splunk Engineer - Infrastructure Operations to support the AOUSC NLS project. The role involves designing, implementing, and operating Splunk Core, Enterprise Security, ITSI, Phantom (SOAR), Splunk Cloud, Splunk On-Call, and multi-site index clustering.

The candidate will monitor health via the DMC, manage ingestion and parsing, optimize searches and dashboards, rotate credentials, and collaborate with the Engineering team on escalations and RCA findings.

Qualifications

• Bachelor's degree or commensurate experience with 10 years in the field, or a Master’s degree with 7+ years.
• Experience designing, implementing, and operating large Splunk environments.
• Strong knowledge of Enterprise Security, ITSI, and SOAR.

Responsibilities

• Design, implement, and operate Splunk Core, ES, ITSI, Phantom/SOAR, Splunk Cloud, and multi-site indexers.
• Monitor health using DMC, track indexing rates, license usage, and search concurrency.
• Troubleshoot data ingestion, parsing, and indexing across forwarders and HEC endpoints.
• Validate sourcetype, timestamps, and field extractions for new data sources.
• Maintain data flow visibility from source to indexer and manage credentials and RBAC.
• Provide end-user support for SPL searches, dashboards, and alerts; document incidents and RCA findings.
• Collaborate with Engineering on escalations, root cause analyses, and deployments.

Skills

Enterprise Security

ITSI

SOAR

Splunk Cloud

Education

Bachelor's degree (or commensurate experience)

Master's degree or higher (related discipline)

Tools

Splunk Core

Splunk Enterprise Security

Splunk ITSI

Splunk Phantom / SOAR

Job description

GovCIO is currently hiring for Senior Splunk Engineer - Infrastructure Operations of Infrastructure Operations to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.

Responsibilities

• Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
• Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.
• Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.
• Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.
• Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.
• Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
• Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
• Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and elevate complex parsing issues to Engineering.
• Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
• Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.
• Maintain data flow visibility from source → forwarder → indexer to confirm data completeness and performance.
• Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
• Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
• Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.
• Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.
• Log and elevate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.
• Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults.
• Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.
• Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.
• Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies.
• Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review.
• Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations.
• Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected.
• Document incidents, RCA findings, and preventive actions for future reference.
• Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications.

Qualifications

Bachelor's with 10 years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 7 years experience

Required Skills And Experience

• Expert skills in Enterprise Security, ITSI, SOAR, and the Slunk product line.
• Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.

Clearance Required: Must be able to obtain and maintain AOUSC Public Trust

Posted Salary Range: USD $105,000.00 - USD $145,000.00 /Yr.