Senior Engineer

Koitecc Solutions; New Jersey; On-site; USD 122,000 - 200,000; Full time

Benefits offered by this job

Discretionary annual bonus

Industry-leading benefits

Paid time off

Job summary

Bank of America is seeking a Principal Splunk Engineer to lead the design, operation, and evolution of a large-scale Splunk Enterprise / Splunk Cloud deployment across security, infrastructure, and applications. The platform ingests multi-terabyte daily data volumes and is central to SOC and threat detection.

Ideal candidates have deep SPL expertise, SmartStore/Indexer Clustering experience, and strong Python/Bash scripting with automation (Ansible, Terraform).

Qualifications

• 5+ years experience administering large Splunk Enterprise or Splunk Cloud environments.
• Strong hands-on knowledge of indexer clustering, search head clustering, and SmartStore integration.
• Deep experience with security log ingestion and SIEM use cases.
• Strong SPL expertise including search optimization and data model acceleration.
• Experience with Linux systems, scripting (Python/Bash), and automation tools (Ansible, Terraform).

Responsibilities

• Architect, operate, and optimize a distributed, large-scale Splunk environment across multiple sites.
• Lead capacity planning, index design, and data retention strategies.
• Maintain high availability and DR across deployments; drive upgrades and hardening.
• Collaborate with SOC and Threat Hunting teams to ensure high-quality log ingestion.
• Establish governance with dashboards, mappings, and documentation.

Skills

Automation

Influence

Result Orientation

Stakeholder Management

Technical Strategy Development

Application Development

Architecture

Business Acumen

Risk Management

Solution Design

Agile Practices

Analytical Thinking

Collaboration

Data Management

Solution Delivery Process

Tools

Splunk

Python

Bash

Ansible

Terraform

GitOps

Job description

Job Description:

At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. We do this by driving Responsible Growth and delivering for our clients, teammates, communities and shareholders every day.

Being a Great Place to Work is core to how we drive Responsible Growth. This includes our commitment to being an inclusive workplace, attracting and developing exceptional talent, supporting our teammates' physical, emotional, and financial wellness, recognizing and rewarding performance, and how we make an impact in the communities we serve.

Bank of America is committed to an in-office culture with specific requirements for office-based attendance and which allows for an appropriate level of flexibility for our teammates and businesses based on role-specific considerations.

At Bank of America, you can build a successful career with opportunities to learn, grow, and make an impact. Join us!

Role Overview:

We are seeking a Principal Splunk Engineer to lead the design, operation, and evolution of our large-scale Splunk Enterprise / Splunk Cloud deployment. The platform ingests multi-terabyte daily data volumes across security, infrastructure, and application domains and is a critical component of our SOC and threat-detection capabilities. The ideal candidate has deep expertise in Splunk architecture, large-scale data onboarding, performance optimization, SmartStore/Indexer Clustering, and security-focused use cases.

Key Responsibilities:

Platform Architecture & Operations:

• Architect, operate, and optimize a distributed, large-scale Splunk environment (indexer clusters, search head clusters, cluster masters, deployment servers, IDM, ADFS/SAML integrations)
• Lead capacity planning, index design, data retention strategies, and SmartStore lifecycle management
• Maintain high availability, scaling, and resilience across multi-site deployments (including DR strategy)
• Drive Splunk version upgrades, app updates, cluster maintenance, and platform hardening

Security Logging & SOC Enablement:

• Collaborate with SOC, Incident Response, and Threat Hunting teams to ensure high-quality security log ingestion
• Onboard and normalize logs from firewalls, EDR, identity platforms, cloud providers, network telemetry, and custom applications
• Develop and optimize detection content: correlation searches, risk-based alerting, data models, macros, lookups, summaries
• Ensure compliance with logging standards (MITRE ATT&CK mapping, CIS/SOC2/ISO27001 logging requirements)

Data Engineering & Observability:

• Build and manage ingestion pipelines, parsing, field extractions, CIM compliance, HEC configurations, and forwarder architecture
• Implement data lifecycle tiers, filtering strategies, routing, and ingestion controls to reduce cost and improve efficiency
• Optimize search performance, knowledge objects, summary indexing, and acceleration strategies

Governance & Best Practices:

• Establish Splunk development standards, dashboards, and naming conventions
• Mentor junior engineers and act as a technical escalation point for the team
• Maintain documentation, operational runbooks, and logging onboarding guidelines
• Partner with Engineering, Cloud, SecOps, and App teams to drive company-wide observability maturity

Required Qualifications:

5+ years experience administering large Splunk Enterprise or Splunk Cloud environments

Strong hands‑on knowledge of:

• Indexer clustering, search head clustering
• SmartStore / S3-compatible object store design
• Universal/heavy forwarder architecture
• Ingest actions, parsing, props/transforms
• KVStore, RBAC, SAML, encryption

Deep experience with security log ingestion and SIEM use cases

Strong SPL expertise, including:

• Search optimization
• Summary indexing / data model acceleration
• CIM mapping and field normalization

Experience with Linux systems engineering, scripting (Python/Bash), and automation frameworks (Ansible, Terraform, GitOps preferred)

Preferred Qualifications:

Splunk certifications (Core Consultant, Enterprise Admin, Enterprise Architect, ES Analyst/ES Admin, or equivalent)

Experience with:

• Enterprise Security (ES)
• SOAR (Phantom or comparable)
• AWS/Azure/GCP cloud logging architectures

Familiarity with high-throughput message brokers (Kafka/FluentD/Cribl)

Background in cybersecurity engineering or threat detection

Skills:

• Automation
• Influence
• Result Orientation
• Stakeholder Management
• Technical Strategy Development
• Application Development
• Architecture
• Business Acumen
• Risk Management
• Solution Design
• Agile Practices
• Analytical Thinking
• Collaboration
• Data Management
• Solution Delivery Process

Shift:

1st shift (United States of America)

Hours Per Week:

40

Pay Transparency details

US - NJ - Jersey City - 101 Hudson St - 101 Hudson (NJ2101)

US - NJ - Pennington - 1300 American Blvd - Hopewell Bldg 3 (NJ2130)

Pay and benefits information

Pay range

$122,000.00 - $200,000.00 annualized salary, offers to be determined based on experience, education and skill set.

Discretionary incentive eligible

This role is eligible to participate in the annual discretionary plan. Employees are eligible for an annual discretionary award based on their overall individual performance results and behaviours, the performance and contributions of their line of business and/or group; and the overall success of the Company.

Benefits

This role is currently benefits eligible. We provide industry-leading benefits, access to paid time off, resources and support to our employees so they can make a genuine impact and contribute to the sustainable growth of our business and the communities we serve.