Senior Linux HPC Systems Engineer

Cadre5; Knoxville (TN); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

Medical insurance

Dental coverage

Vision coverage

401K match

PTO 15 days

Holidays 10

Job summary

Cadre5 is seeking a Senior Linux HPC Systems Engineer to join the Oak Ridge National Laboratory ecosystem. You will champion HPC and clustered computing services for researchers handling large data sets and code development, ensuring production systems’ availability, performance, and security.

The role emphasizes automation, monitoring, and collaboration with program POCs to install and tune scientific toolsets. A strong Linux background and DOE clearance willingness are essential.

Qualifications

• BS degree in CS/CE/IT/science with 8–12 years of proven experience.
• Five or more years with managing UNIX/Linux Systems.
• Three or more years of proven experience with configuration management and automation tools such as Git, Jenkins, Ansible, or Puppet.
• Moderate proficiency in at least one scripting language such as Bash, Python, or others.
• Experience performing advanced troubleshooting and system administration with Linux Servers.
• Ability to obtain and maintain a Department of Energy "Q" clearance (US Citizenship).

Responsibilities

• Advocate and promote HPC and clustered computing services to researchers who process large data sets.
• Ensure the availability, performance, scalability, and security of production systems.
• Leverage automation and monitoring solutions to minimize day-to-day maintenance and optimize system performance.
• Collaborate with technical POCs to install and tune the performance of scientific toolsets.
• Optimize workflows and monitoring for 24/7 operations to reduce off-hours support.

Skills

UNIX/Linux administration

Scripting: Bash Python

Automation tools: Git Jenkins Ansible

Cluster computing

HPC environments

Education

BS degree in CS/CE/IT/science

Tools

SLURM

Warewulf

Cobbler

PXE boot

Bright

Lustre

GPFS

NVIDIA GPUs

Job description

Location: Knoxville, TN

Job Id:534

# of Openings:1

Founded in 1999 in the beautiful Smoky Mountains of East Tennessee, Cadre5 provides innovative technical solutions to our customers locally and nationally. Our Cadre5 Lab Partners division has partnered with the Emerging Technologies & Computing (ETAC) Group in the Research Computing Support Division (RCSD) of the Information Technology Services Directorate (ITSD) at Oak Ridge National Laboratory (ORNL) to recruit a qualified Senior Linux HPC Systems Engineer. The Emerging Technology Computational Group facilitates ORNL goals through HPC systems engineering, integration, and support for the research community at ORNL. By providing design, deployment, optimization, monitoring, and tooling support across multiple clustered infrastructures, they facilitate Lab-wide R&D projects. Their HPC clusters range in scope from just a handful of nodes to over fifty thousand cores.

ORNL delivers scientific discoveries and technical breakthroughs needed to realize solutions in energy and national security and provides economic benefit to the nation. This premier research institution located near Knoxville in Oak Ridge, TN, addresses national needs through impactful research and world-leading research centers.

This is a full-time, permanent position that requires on-site work in Oak Ridge, TN.

Why Cadre5?

• Working with highly talented team members
• Excellent medical insurance, i ncluding employer-paid benefits

Job Responsibilities:

• Advocate and promote HPC and clustered computing services to researchers who process large data sets and/or develop code as a part of their project.
• Ensure the availability, performance, scalability, and security of production systems.
• Leverage automation and monitoring solutions that minimize our day-to-day maintenance and scout opportunities to optimize system management practices or system performance.
• Collaborate with technical POCs for the programs that we support to install and help tune the performance of various scientific toolsets.
• Optimize both workflows and monitoring solutions to take advantage of our 24/7 operations staff, which significantly reduces the need for off-hours support. We use Email, Jira, Confluence, Teams, Slack, and other collaboration solutions to stay in contact.

Basic Qualifications:

• A BS degree in computer science, computer engineering, information technology, business, science, or a related field of study and eight (8) to twelve (12) years of proven experience is required. An overall combination of equivalent experience may be considered.
• Five (5) or more years with managing UNIX/Linux Systems.
• Three (3) or years of proven experience with configuration management and automation tools such as Git, Jenkins, Ansible, or Puppet.
• Moderate proficiency in at least one scripting language such as Bash, Python, or others.
• Experience performing advanced troubleshooting and system administration with Linux Servers.
• A strong desire to innovate and identify new technologies and opportunities and be able to communicate the potential benefits of those choices to others within the team and our research partners.
• A collaborative and upbeat approach to thrive on the opportunity to build trust and credibility, and ultimately become a trusted advisor to our research teams.
• The ability to obtain and maintain a Department of Energy "Q" clearance is required. This requires US Citizenship.

Preferred Qualifications:

• Solid understanding of multiple operating systems and cluster technologies.
• Experience with Centos/RHEL, Ubuntu, VMware
• Understanding of HPC platforms to support users with SLURM job submissions and troubleshooting.
• Experience building and running containerized applications in an HPC environment.
• Experience with multiple deployment mechanisms like Diskless, Warewulf, and traditional deployment (cobbler, PXEboot, and/or Bright).
• Experience managing systems utilizing GPU (NVIDIA and AMD) clusters for AI/ML and/or image processing.
• Knowledge of networking fundamentals including TCP/IP, traffic analysis, common protocols, and network diagnostics.
• Experience with Infiniband networks and diagnostics.
• Extensive experience with High Performance Parallel File Systems (Lustre, WEKA, GPFS, etc).
• Experience with performance and diagnostic tools for benchmarking, analysis and tuning of systems, networking, and storage.
• Experience with Grafana, CheckMK, Nagios, Zabbix, SolarWinds, Ganglia, or other network and device monitoring systems.
• Previous experience working in a government, scientific or other highly technical environment.
• Good documentation skills, including ability to prepare simple documentation web pages.

Cadre5 offers excellent pay and benefits, to include full medical, dental, and vision coverage coupled with 401K match, 15 days PTO, and 10 holidays.

Cadre5 is an equal opportunity employer. All qualified applicants, including individuals with disabilities and protected veterans, are encouraged to apply. Cadre5 is an E-Verify Employer.