24/7 HPC Site Reliability Engineer - Automation & Monitoring

Energy Jobline ZR; Berkeley (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

NERSC is seeking a Site Reliability Engineer to join the Operations Technology Group. The role focuses on monitoring the HPC facility, triaging alerts, and building automation to keep systems reliable and secure for scientific users.

You will work in a 24/7 onsite team, with shifts for overnight monitoring. The ideal candidate has strong Linux, scripting, and debugging skills, plus experience with Kubernetes, Prometheus, and ITSM practices.

Qualifications

• Experience working in or willingness to work within a 24/7 onsite team environment.
• Experience with Linux shell and command-line environments (SSH).
• Experience with programming languages or scripting (C, C++, Python, Perl, Java).
• Motivated self-starter who can learn Kubernetes, Prometheus/VictoriaMetrics, Alertmanager and automation tools.
• Familiarity with ServiceNow implementation and ITIL/ITSM practices.

Responsibilities

• Works an onsite 5-day weekly schedule for overnight monitoring of the HPC facility.
• Review and respond to alerts from systems, storage, network, and data center facilities.
• Create solutions to improve processes and automate routine responses.
• Identify issues and propose improvements for monitoring and triage.
• Develop and implement customized service management in ServiceNow.
• Respond to alerts to ensure 24/7 data collection and diagnosis readiness.
• Develop and maintain monitoring tools in collaboration with the Operations Team.
• Create software to provide alerts from HPC APIs into the monitoring pipeline.
• Build and maintain application/tool configurations for reliability.
• Collaborate with other groups to clarify communications and workflows.
• Coordinate center-wide maintenance and manage diagnostic software during maintenance.
• Perform physical/logical walkthroughs of the data center for health and cooling.
• Provide accurate information in trouble-ticketing systems for outages and maintenance.
• Handle problems of diverse scope requiring data analysis and judgment.
• Demonstrate good judgment in selecting methods for solutions.
• Resolve complex issues with in-depth evaluation of factors.

Skills

Linux shell

SSH

C/C++

Python

Perl

Java

Kubernetes

Prometheus/VictoriaMetrics

Alertmanager

Automation tooling

Tools

ServiceNow

ITSM

Job description

NERSC is seeking a Site Reliability Engineer to join the Operations Technology Group. The role focuses on monitoring the HPC facility, triaging alerts, and building automation to keep systems reliable and secure for scientific users.

You will work in a 24/7 onsite team, with shifts for overnight monitoring. The ideal candidate has strong Linux, scripting, and debugging skills, plus experience with Kubernetes, Prometheus, and ITSM practices.