Principal Splunk Engineer

Leidos; United States; Remote; USD 126,000 - 228,000; Full time

Job summary

An innovative firm is seeking a Principal Splunk Engineer to spearhead its Enterprise Observability initiatives. This role involves architecting and evolving complex Splunk infrastructures, ensuring operational excellence while managing vast data streams. The ideal candidate will possess deep expertise in Splunk, cloud technologies, and advanced automation. Join a forward-thinking team that values innovation and is committed to shaping the future of enterprise observability. With a fully remote opportunity, this position offers flexibility and the chance to make a significant impact in a rapidly evolving field.

Qualifications

• 12+ years of IT experience with 6+ years in Splunk Engineering.
• Strong scripting skills in Python and JavaScript are essential.
• Expertise in AWS and Azure cloud technologies is required.

Responsibilities

• Lead the architecture and implementation of Splunk infrastructure.
• Drive innovation in observability platforms and automation solutions.
• Oversee performance analysis, upgrades, and integrations.

Skills

Splunk Engineering

Linux Management

Python Programming

JavaScript Programming

REST API Integration

AWS Cloud Technologies

Azure Cloud Technologies

Data Normalization

Advanced Troubleshooting

Agile Methodologies

Education

Bachelor's Degree in IT

Master's Degree in IT

Tools

Splunk

Cribl Stream

Docker

Kubernetes

Ansible

Job description

Are you ready to make an impact?

The Leidos CIO team, within the Digital Modernization sector, is seeking a visionary Principal Splunk Engineer to lead and shape our growing Enterprise Observability initiatives as a core member of our CIO Chief Data and Analytics Office Performance Management team. We are significantly expanding our global Splunk infrastructure and enhancing our capabilities over the next three years. This critical role demands a seasoned expert who can architect, implement, and continuously evolve complex, enterprise-scale Splunk and Cribl environments, setting new standards for operational excellence.

In this role, you will serve as the technical cornerstone, driving innovation across the full lifecycle of our observability platforms: requirements definition; strategic architecture planning; advanced performance analysis and tuning; major system upgrades and expansions; development of cutting-edge capabilities; automation and orchestration leadership; sophisticated troubleshooting; security integrations; and overseeing comprehensive documentation and governance.

Location: This role offers a 100% remote working opportunity from any U.S. based location, providing flexibility to attract top global talent dedicated to shaping the future of enterprise observability at Leidos.

The Leidos Splunk infrastructure is expansive, complex, and growing:

• Daily ingest of 8+ TB across 600+ users globally

• Robust physical and virtual on-premises infrastructure:

  • Multi-site indexer cluster (60+ nodes)

  • Multiple search head clusters (20+ nodes)

  • Diverse deployment servers and standalone instances (10+ nodes)

• Cribl Stream intermediate processing layer:

  • 30+ nodes across three continents, managing vast syslog data streams

• Infrastructure supporting 50,000+ unique hosts

• Hundreds of ODBC connections, custom/modular inputs, and integration points

• Thousands of dashboards, reports, saved searches, lookups, and KVstores

• Over 200 apps including custom-built and vendor-supplied

• Comprehensive cloud integrations (AWS, Azure), Kubernetes integrations, and sophisticated disaster recovery frameworks

• Adherence to stringent federal and financial governance requirements

Additionally, the CIO Performance Management team is aggressively advancing in strategic areas:

• Splunk ITSI: Leading service decomposition and system-of-systems integrations

• Advanced Orchestration and Automation: Deep integration with ServiceNow and other enterprise management platforms

• Next-Generation Interface Development: Creating sophisticated, user-centric dashboards beyond conventional Splunk offerings

• Applied Machine Learning: Utilizing statistical models to deliver proactive and intelligent monitoring solutions

If you are a recognized expert with deep experience architecting, building, and evolving large-scale Splunk infrastructures and possess proficiency in any of these advanced strategic areas, we'd be excited to connect with you!

Required Qualifications:

• Bachelor's degree with a minimum of 12 years of relevant IT experience; Master's and 10 years, Associate and 14 years or additional experience may be considered in lieu of a degree

• 6+ years as a senior-level Splunk Engineer

• 8+ years managing complex Linux-based infrastructures

• US Citizenship required

• Proven expertise architecting, deploying, and enhancing Splunk infrastructures handling multi-terabyte daily ingests

• Advanced proficiency with Splunk Apps, Add-ons, props, transforms, and Knowledge Objects

• Extensive knowledge of Splunk's Common Information Model and data normalization

• Deep expertise with Cribl Stream data processing solutions

• Proven track record integrating Splunk ITSI for enterprise-scale service decomposition

• Strong scripting/programming skills, particularly in Python and JavaScript

• Solid experience with REST API integrations and application interface development

• Mastery of regular expressions and advanced troubleshooting methodologies

• Expert-level proficiency with AWS and Azure cloud technologies

• Demonstrated leadership and project management abilities, including agile methodologies

• Working knowledge of ITIL Change & Configuration Management

Desired Qualifications:

• Ability to obtain Secret security clearance as per program requirement

• Advanced experience with Kafka for secure data streaming platforms

• Hands-on experience implementing Splunk-based machine learning solutions

• Prior experience leading Splunk Cloud migrations

• Practical experience applying data analytics for strategic decision-making

• Expertise in workflow automation tools, including scripting and low-code/no-code solutions

• Proficiency with containerization technologies such as Docker and Kubernetes

• Experience with infrastructure automation using Ansible

• Familiarity with SQL/ODBC interfaces and data integrations

• Broader experience with system/network monitoring and management tools

• Completion or ongoing certification as a Splunk Admin/Architect

• Master's degree in Information Technology, Computer Science, or related field preferred

Original Posting: May 6, 2025

For U.S. Positions: While subject to change based on business needs, Leidos reasonably anticipates that this job requisition will remain open for at least 3 days with an anticipated close date of no earlier than 3 days after the original posting date as listed above.

Pay Range: Pay Range $126,100.00 - $227,950.00

The Leidos pay range for this job level is a general guideline onlyand not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.