Monitoring (Technical Splunk Engineer)This position is for a Logging Engineer. As a technical Splunk Engineer, you'll have the opportunity to manage the complex challenges of scale within a logging and metrics environment, while using your expertise in development, complexity analysis and large-scale system design. The engineer must adopt and support a culture of diversity, intellectual curiosity, problem solving and openness.
Key Responsibilities Include:
- Responsible for the effective prioritization of tickets and timely resolution of issues
- Responsible for managing communications of outages, service impacts, and all other Splunk infrastructure-related events within the SLA defined by the Mission Support organization
- Responsible for monitoring system health, ensuring proactive measures are applied, or issues are raised to the engineering team for resolution
- Proactively monitor the Splunk infrastructure and follow an iterative, continuous improvement model to ensure that the system is healthy
- Perform after-action reports and root-cause analysis reviews with the architecture and engineering team
- Monitor the monitoring console and develop new checks or continuously fine-tune the checks and alerts
- Perform routine software upgrades and minor releases
- Adhere to the change control process and ensure that all work is documented
- Regularly audit the authentication and authorization systems
- Build a Splunk Hygiene Program that focuses on optimal searches and configuration
- Monitor Splunk knowledge objects, DMAs, and user queries to ensure optimum performance
Required Qualifications
- 3+ years of Splunk experience
- 5+ years of Linux experience
- 3+ years of experience with scripting languages such as Bash, PowerShell, Python, Shell, JavaScript, Ansible
Desired Qualifications
- Splunk SmartStore (configuration, support, and implementation) is highly desired
- Ability to perform full stack troubleshooting
- Experience in managing IT infrastructure for a medium-to-large organization
- Relationship management experience with organizational partners and customers
- Practical knowledge of ITIL application processes and implementation
- Knowledge and understanding of cloud computing
- 5+ years of experience troubleshooting environments across the entire architecture (i.e. applications to infrastructure)
- Knowledge and understanding of technology support: troubleshooting environments across the entire architecture (applications to infrastructure)
- Experience with Ansible automation tool
- Ability to manage large complex cross-matrix organization that includes: multiple business, technology and risk disciplines with direct leadership; the development of the strategy for information security; and implementation of roadmaps for the Enterprise
- Ability to manage multiple enterprise-wide: technology infrastructure, service components, or end-to-end distributive infrastructure activities
- Knowledge or experience of containerization related management stacks with Docker, Swarm or Kubernetes
- Experience with continuous integration and continuous deployment
- Experience with a customer service mindset and a focus on continuous improvement
- BS/BA degree or higher
- Experience with Splunk premium products a plus – ES, ITSI, UBA, MLTK, etc.
- Splunk query, reporting, and dashboard skills with particular emphasis on reporting on infrastructure and application performance and capacity
- Splunk administration skills in an enterprise environment ingesting >10TB daily
- Experience administering a mid-large scale Splunk deployment(s)
- Experience engineering and maintaining the enterprise strategic implementation of Splunk deployments highly preferred
- Experience with developing and deploying Regular Expressions (REGEX)
- Certified Splunk Architect or Engineer highly preferred
- Experience with Cribl Stream is a plus