IT Resident Engineer -Splunk maintenance

WSH Experts Pte Ltd; Singapore; On-site; SGD 60,000 - 80,000; Full time

Job summary

A technology consulting firm in Singapore is seeking a Resident Engineer with Splunk expertise to manage and monitor Splunk systems. Responsibilities include troubleshooting, ensuring system health, and collaborating with the Authority’s SOC for threat detection. Candidates should have at least 3 years of experience with Splunk and hold a relevant certification. This role involves critical maintenance work, including off-hours responsibilities, and offers compensatory off-days.

Qualifications

• At least 3 years of experience working on Splunk systems.
• Possess Splunk Enterprise Certified Admin certifications or equivalent.

Responsibilities

• Perform checks to ensure Splunk services are operating correctly.
• Maintain and monitor Splunk infrastructure for uptime.
• Generate reports and troubleshoot issues with Splunk systems.

Skills

Experience with Splunk systems

Troubleshooting and monitoring

Incident reporting

Working with SOC

Education

Splunk Enterprise Certified Admin certification

Job description

Job Description

• Perform checks and troubleshoot, if necessary, to ensure the Splunk services are running as intended for all environments.
• Maintain and monitor Splunk infrastructure (Search Heads, Indexers, Forwarders, Deployment Server, Cluster Master, etc.).
• Ensure uptime and system health via monitoring, tuning, and log analysis (including introspection, metrics logs).
• Manage indexing performance and storage usage: data retention, index lifecycle, bucket management.
• Generate and check reports from the system to ensure the system and agents are working as intended.
• Perform checks and troubleshoot if necessary, to ensure that the Splunk forwarders (agents) are working and can pipe logs back to Splunk systems.
• Perform checks and troubleshoot if necessary, to ensure the Splunk systems can receive logs from sources such as CloudWatch or syslogs servers.
• Integrate Splunk with the Authority’s systems and processes to perform real‑time monitoring and alert when Splunk infrastructure is not working well, so that issues can be attended to early (eg. log breaks, disconnected agents, search‑head hung from insufficient resources, etc).
• Fine‑tune Splunk rules according to the Authority’s request.
• Perform parser validation or write new custom parser according to the Authority’s request.
• Work closely with the Authority’s SOC to ensure Splunk supports threat detection, auditing, and incident response use cases.
• Change the passwords for all privileged and service accounts for the Splunk systems regularly.
• Ensure the Splunk systems are working as intended during the Authority’s periodic BCP and DR exercises.
• Investigate problems and provide assistance to triage issues.
• Correct defects in the system, including temporary corrections or workarounds until permanent fixes or updates are available.
• Prepare incident reports including root‑cause analysis and necessary resolution.
• Track and report issues, support cases and incident resolutions on a weekly basis.
• The Resident Engineer shall perform critical high‑risk works during maintenance windows specified by the Authority, which may be off‑office hours or during weekends, at no additional cost to the Authority. The Authority will compensate with off‑days for work done over non‑working days.
• The Resident Engineer shall be responsible for all corrective and preventive maintenance of the Splunk systems in all environments.
• The Resident Engineer shall remediate all vulnerabilities or penetration test findings pertaining to the Splunk systems.
• The Resident Engineer can raise tickets to Splunk principal for support and queries.
• System monitoring & system changes.

Job Requirement

• At least 3 years of experience working on Splunk systems.
• Possess Splunk Enterprise Certified Admin certifications or equivalent.
• If the requirement matches your profile, kindly share your updated CV/resume to Aparna at [e-mail].