HPC Operations Engineer

Lawrence Livermore National Laboratory; Livermore (CA); On-site; USD 122,000 - 186,000; Full time

Job summary

Lawrence Livermore National Laboratory is seeking an HPC Operations Engineer to build and maintain monitoring, automation, and diagnostic tools for LC clusters, storage, and data center facilities. You will work with system administrators, developers, and facilities staff to improve reliability and performance in an on-site role, supporting multiple shifts as required.

Responsibilities include deploying and validating new HPC systems, maintaining runbooks, and assisting end users while

Qualifications

• Proven knowledge of Linux/UNIX and enterprise data storage concepts.
• Experience with programming/scripting languages (C/C++, Rust, Go, Python, or Bash).
• Familiarity with version control and configuration management systems (Git, Ansible, CFEngine).
• Basic understanding of monitoring, alerting, and incident management practices.

Responsibilities

• Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for LC HPC clusters, storage, networks, and data center facilities.
• Create and maintain software tools and processes to streamline alarm response, diagnostics, and routine operations.
• Integrate data from building management and facility systems with HPC monitoring for a unified view.
• Collaborate with staff to diagnose and resolve hardware, software, network, and facility issues of limited complexity.

Skills

Linux/UNIX

Programming languages

Troubleshooting

Communication

Education

Bachelor's degree in computer science or related field

Tools

Git

Ansible

CFEngine

Grafana

Prometheus

Splunk

Nagios

Docker

Kubernetes

Job description

Job Description

We have an opening for an HPC Operations Engineer to develop, deploy, and maintain the tools, automation, and processes that keep Livermore Computing high-performance computing systems available and performing at high levels of reliability and performance. You will work in a challenging, team-oriented environment, applying technical knowledge of HPC systems and software development to engineer monitoring, alerting, and diagnostic solutions for Livermore Computing’s (LC) high-performance computing clusters, storage systems, networks, and data center facilities. You will contribute to technical projects as a member of a technical team, and collaborate with system administrators, software developers, and facilities engineers to improve the reliability and efficiency of LC operations. This position is in the Livermore Computing Division within the Computing Directorate.

This position will be filled at either level based on knowledge and related experience as assessed by the hiring team. Additional job responsibilities (outlined below) will be assigned if hired at the higher level.

This position requires full-time on-site presence due to the nature of the work.

Join us and make YOUR mark on the World!

Lawrence Livermore National Laboratory (LLNL) has turned bold ideas into world-changing impact advancing science and technology to strengthen U.S. security and promote global stability.

Our mission spans four critical national security areas nuclear deterrence, threat preparedness, energy security, and multi-domain defense empowering teams to take on the toughest challenges of today and tomorrow. With a culture built on innovation and operational excellence, LLNL is a place where your expertise can make a real impact.

You will

• Develop, deploy, configure, and maintain monitoring, alerting, and observability tools for LC HPC clusters, storage systems, high-speed networks, and data center facility infrastructure.
• Create and maintain software tools and processes that streamline alarm response, diagnostics, and routine operational tasks using established methods and procedures.
• Configure and integrate data from building management and facility systems, such as power and cooling, with HPC system monitoring to provide a unified operational view under established technical direction.
• Share alarm response and monitoring duties with operations staff, and analyze, diagnose, and resolve hardware, software, network, and facility problems of limited complexity, determine root cause, and implement corrective actions in collaboration with system administrators, vendors, and facilities staff.
• Participate in the deployment, testing, and validation of new HPC systems and components, and the decommissioning of retired systems.
• Develop and maintain technical documentation, procedures, and runbooks for operational tools and processes.
• Respond to customer problems and questions, working with Hotline, system administration, and other LC groups to improve the quality of service for end users.
• Perform other duties as assigned.

Additional job responsibilities, at the SES.2 level

• Design and develop moderately complex software tools and automations that improve the operation, reliability, and efficiency of LC computing and facility systems.
• Analyze system, alarm, and incident data to identify trends, and implement moderately complex strategies to improve monitoring coverage, alarm accuracy, and operational processes.
• Lead discrete technical projects or work packages, such as the deployment, validation, and decommissioning of HPC systems, coordinating tasks with system administration, facilities, and other LC groups.
• Diagnose and resolve moderately complex to complex system and facility problems and serve as a technical escalation resource.
• Develop and deliver training and provide guidance and informal mentoring to less-experienced staff.
• Independently prioritize assigned work across multiple concurrent operational and project tasks.

Qualifications

• Ability to secure and maintain a U.S. DOE Q-level security clearance, which requires U.S. citizenship.
• Bachelor’s degree in computer science, computer engineering, or related field or the equivalent combination of education and related experience.
• Fundamental knowledge of and experience with Linux/UNIX systems and enterprise data storage, including installation, configuration, networking, RAID concepts, system logs, and troubleshooting.
• Understanding of programming and scripting languages, such as C/C++, Rust, Go, Python, or bash/csh/ksh.
• Experience with version control and configuration management systems, such as Git, Ansible, or CFEngine.
• Fundamental knowledge of system monitoring, alerting, and incident management practices and tools.
• Sufficient verbal and written communication and interpersonal skills necessary to collaborate effectively with system administrators, software developers, facilities staff, LC staff, and end users, and to present and explain technical information.
• Ability to work all shifts, including Owl shift (12am – 8am), weekends, and holidays.

Additional qualifications at the SES.2 level

• Comprehensive knowledge of HPC environments and technologies, such as Linux clusters, InfiniBand, Slurm, Lustre, and archival storage systems.
• Proficient experience developing software tools and automation in C/C++, Python, or similar languages, and broad application of technical principles to resolve moderately complex system, network, and facility issues.
• Experience leading technical work packages or discrete projects and mentoring less-experienced staff, and ability to work with limited direction in a dynamic environment with competing priorities.
• Relevant previous experience and demonstrated ability to perform independent technical work.

Qualifications We Desire

• Experience with monitoring and observability tools, such as Grafana, Prometheus, Splunk, or Nagios.
• Experience with data center utility infrastructure and building management systems, including power distribution, water distribution, and cooling distribution units supporting HPC environments.
• Experience with virtual machines, container technologies, and orchestration, such as Docker, Podman, or Kubernetes.
• Experience working in a Closed Area or other secure computing environment.

Pay Range

$121,830 - $185,544 annually

$121,830 - $154,500 annually for SES.1 level

$146,340 - $185,544 annually for SES.2 level

This is the lowest to highest salary we in good faith believe we would pay for this role at the time of this posting; pay will not be below any applicable local minimum wage. An employee’s position within the salary range will be based on several factors including, but not limited to, specific competencies, relevant education, qualifications, certifications, experience, skills, seniority, geographic location, performance, and business or organizational needs.

Additional Information

#LI-Onsite

Position Information

This is a Career Indefinite position, open to Lab employees and external candidates.

Why Lawrence Livermore National Laboratory?

• Included in 2026Best Places to Work by Glassdoor!
• Flexible schedules (*depending on project needs)

This position requires a Department of Energy (DOE) Q-level clearance.If you are selected, wewill initiate a Federal background investigation to determine if youmeet eligibility requirements for access to classified information or matter. Also, all L or Q cleared employees are subject to random drug testing. Q-level clearance requires U.S. citizenship.

Pre-Employment Drug Test

External applicant(s) selected for this position must pass a post-offer, pre-employment drug test. This includes testing for use of marijuana as Federal Law applies to us as a Federal Contractor.

Wireless and Medical Devices

Per the Department of Energy (DOE), Lawrence Livermore National Laboratory must meet certain restrictions with the useand/or possession ofmobile devices in Limited Areas. Depending on your job duties, you may be required to work in a Limited Area whereyou are not permitted to have a personal and/or laboratory mobile devicein your possession. This includes, but not limited to cell phones, tablets, fitness devices, wireless headphones, and other Bluetooth/wireless enabled devices.

Ifyou useamedical device, whichpairs with a mobile device,you must still follow the rules concerningthe mobile device in individual sections within Limited Areas. Sensitive Compartmented Information Facilities requireseparate approval. Hearing aids without wireless capabilities or wireless that has been disabled are allowed in Limited Areas, Secure Space and Transit/Buffer Space within buildings.

We are an equal opportunity employer that is committed to providing all with a work environment free of discrimination and harassment. All qualified applicants will receive consideration for employment without regard to race, color, religion, marital status, national origin, ancestry, sex, sexual orientation, gender identity, disability, medical condition, pregnancy, protected veteran status, age, citizenship, or any other characteristic protected by applicable laws.

Reasonable Accommodation

Our goal is to create an accessible and inclusive experience for all candidates applying and interviewing at the Laboratory. If you need a reasonable accommodation during the application or the recruiting process, please use our online form to submit a request.

CaliforniaPrivacy Notice

The California Consumer Privacy Act (CCPA) grants privacy rights to all California residents. The law also entitlesjob applicants, employees, and non-employee workers to be notified of what personal information LLNL collects and for what purpose. The Employee Privacy Notice can be accessed here .

Lawrence Livermore National Laboratory 92K subscribers