Sr. High-Performance Computing Engineer - 141797

UC San Diego

San Diego (CA)

On-site

USD 150,000 - 210,000

Full time

10 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

UC San Diego is seeking a Sr. HPC Engineer to lead the technical operation of the AWARE high-performance computing system.

You will manage compute and GPU nodes, networks, storage, and the job scheduler, collaborating with researchers and cyberinfrastructure leads to ensure reliability for near real-time forecasting and research workloads. The role involves planning future upgrades, mentoring staff, and maintaining thorough documentation while troubleshooting complex multi-layer systems across

Qualifications

  • Advanced knowledge of systems integration deploying complex systems integration solutions.
  • Advanced skills associated with system integration design, modification, implementation and deployment in a complex environment.
  • Demonstrated ability to develop complex conversion and system implementation plans.
  • Demonstrated ability to understand functional needs of the campus / medical center and / or UC system and how systems can support those needs.
  • Demonstrated ability to interface with senior level technical and non-technical management on a regular basis.
  • Demonstrated ability to create software specifications.
  • Experience with identification and use of code libraries and open-source forums.
  • Experience with planning for deployment and creation of feedback mechanisms.
  • Experience developing and implementing complex project plans.
  • Experience researching and evaluating new technology and solutions for complex environments.
  • Ability to impart knowledge to senior level technical and non-technical audiences and to influence outcomes to meet objectives.
  • Demonstrated ability to identify software and hardware interdependences in a complex multi-platform environment.
  • Demonstrated ability to develop solutions for complex business and systems / custom / ad-hoc solutions.
  • Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.
  • Experience with complex multi-platform troubleshooting in a multi-platform environment.
  • Demonstrated ability to plan and execute the technical aspects of deploying, upgrading, and decommissioning storage systems.
  • Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.
  • Experience with complex multi-platform troubleshooting in a multi-platform environment.
  • Demonstrated knowledge of TCP/IP and Infiniband networking, including sockets, VLANs, and firewalls.
  • Proven experience designing, deploying, and operating large-scale (petabyte-class) high-performance parallel and distributed file systems (e.g., Lustre, Ceph, BeeGFS, GPFS), as well as enterprise and local file systems (e.g., NFS, ZFS, ext4, XFS) in Linux-based environments, including troubleshooting and performance tuning.
  • Expert knowledge and experience with large data storage arrays (more than100TB), and methods to administer, monitor and upgrade.
  • Proven experience administering and supporting large-scale HPC clusters or other distributed POSIX (Linux) systems, including advanced knowledge of Linux system administration, primarily Red Hat and its derivatives (e.g., Rocky Linux).
  • In depth experience in cluster management tasks including deployment, configuration, and troubleshooting of compute nodes, management nodes, network switches, and file servers.
  • Demonstrated experience with scripting and automation using languages such as Bash and Python; use of configuration management tools (e.g., Ansible, CFEngine); and version control systems (e.g., Git) to manage and maintain system configurations and infrastructure.
  • Advanced knowledge of HPC middleware stack including cluster management tools including Slurm, PBS, HPCM, and Bright Cluster Manager.
  • Demonstrated experience evaluating new storage system technologies.

Responsibilities

  • Lead the technical operation of the HPC system (AWARE) including administration, maintenance, upgrades, deployments, and troubleshooting.
  • Ensure high availability for time-sensitive near realtime forecasting and research workloads; plan changes carefully to minimize disruption.
  • Manage compute and GPU nodes, networks, parallel storage, job scheduling, authentication, monitoring and data-transfer services.
  • Install and integrate new hardware/software; maintain configurations and automation; monitor performance and capacity; coordinate with campus teams and vendors.
  • Troubleshoot jobs, MPI applications, software environments, containers, memory and I/O problems, and CPU/GPU performance across multiple layers.
  • Assist in planning future additions/upgrades including gathering requirements, benchmarking, procurements, and deployment planning with minimal downtime.
  • Share knowledge with staff; mentor, explain HPC concepts, review work, and improve operational documentation.
  • Plan and execute deployment, upgrades, and decommissioning of storage systems; work with multi-platform environments.

Skills

Linux system administration
Slurm workload manager
BeegFS / Lustre storage
MPI & GPU computing
Python scripting
Automation (Ansible, -xcat)
Networking (TCP/IP, InfiniBand)
Performance tuning
Documentation & mentoring

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Slurm
BeeGFS
Lustre
ZFS
GIT
Nagios / Prometheus / Telegraf

Job description

DESCRIPTION

The Sr. High-Performance Computing (HPC) Engineer will be responsible for the technical operation of AWARE, CW3E's high-performance computing system. Working with the CW3E Research Cyberinfrastructure Lead, this person will take the lead on system administration, maintenance, upgrades, deployments, and troubleshooting. AWARE supports both time-sensitive "near realtime" operational forecasting and research, so system availability and careful planning of changes are important parts of the job.

The work covers the full HPC environment, including compute and GPU nodes, Ethernet and InfiniBand networks, parallel storage, cluster management, job scheduling, authentication, monitoring, and data-transfer services. The engineer will install and integrate new hardware and software, maintain system configurations and automation, monitor performance and capacity, and respond to production problems. They will also coordinate with campus teams and vendors when outside support is needed and maintain documentation for system changes, procedures, and recovery.

A significant part of the job involves working directly with researchers and operational workflow managers. The engineer will help troubleshoot jobs, MPI applications, software environments, containers, memory and I/O problems, and CPU/GPU performance. This requires being able to determine whether an issue is coming from the application, software environment, scheduler, network, storage system, or hardware and then work with the appropriate people to resolve it.

The engineer will help the CW3E Research Cyberinfrastructure Lead plan future additions and upgrades to AWARE. This may include gathering requirements, comparing technical options, developing benchmarks, working through procurements, and planning deployments with as little disruption as possible.

As a senior member of the group, the engineer will also be expected to share knowledge with other staff. This includes mentoring team members, explaining HPC concepts, reviewing technical work, and improving the documentation needed to operate and support the system.

The position requires applicable experience supporting HPC, research computing, or large-scale Linux systems. Experience with technologies used at CW3E, such as Slurm, InfiniBand, BeeGFS or Lustre, ZFS, MPI, GPU computing, configuration-management platforms (BCM, xcat), and system-monitoring & metric tools (telegraf, influxdb, nagios), is desired but not required. Experience with comparable systems and the ability to learn and troubleshoot unfamiliar technologies will also be considered.

QUALIFICATIONS
  • Advanced knowledge of systems integration deploying complex systems integration solutions.

  • Advanced skills associated with system integration design, modification, implementation and deployment in a complex environment.

  • Demonstrated ability to develop complex conversion and system implementation plans.

  • Demonstrated ability to understand functional needs of the campus / medical center and / or UC system and how systems can support those needs.

  • Demonstrated ability to interface with senior level technical and non-technical management on a regular basis.

  • Demonstrated ability to create software specifications.

  • Experience with identification and use of code libraries and open-source forums.

  • Experience with planning for deployment and creation of feedback mechanisms.

  • Experience developing and implementing complex project plans.

  • Experience researching and evaluating new technology and solutions for complex environments.

  • Ability to impart knowledge to senior level technical and non-technical audiences and to influence outcomes to meet objectives.

  • Demonstrated ability to identify software and hardware interdependences in a complex multi-platform environment.

  • Demonstrated ability to develop solutions for complex business and systems / custom / ad-hoc solutions.

  • Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.

  • Experience with complex multi-platform troubleshooting in a multi-platform environment.

  • Demonstrated ability to plan and execute the technical aspects of deploying, upgrading, and decommissioning storage systems.

  • Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.

  • Experience with complex multi-platform troubleshooting in a multi-platform environment.

  • Demonstrated knowledge of TCP/IP and Infiniband networking, including sockets, VLANs, and firewalls.

  • Proven experience designing, deploying, and operating large-scale (petabyte-class) high-performance parallel and distributed file systems (e.g., Lustre, Ceph, BeeGFS, GPFS), as well as enterprise and local file systems (e.g., NFS, ZFS, ext4, XFS) in Linux-based environments, including troubleshooting and performance tuning.

  • Expert knowledge and experience with large data storage arrays (more than100TB), and methods to administer, monitor and upgrade.

  • Proven experience administering and supporting large-scale HPC clusters or other distributed POSIX (Linux) systems, including advanced knowledge of Linux system administration, primarily Red Hat and its derivatives (e.g., Rocky Linux).

  • In depth experience in cluster management tasks including deployment, configuration, and troubleshooting of compute nodes, management nodes, network switches, and file servers.

  • Demonstrated experience with scripting and automation using languages such as Bash and Python; use of configuration management tools (e.g., Ansible, CFEngine); and version control systems (e.g., Git) to manage and maintain system configurations and infrastructure.

  • Advanced knowledge of HPC middleware stack including cluster management tools, job schedulers and resources managers. Examples include: Slurm, PBS, HPCM, and Bright Cluster Manager.

  • Demonstrated experience evaluating new storage system technologies.

SPECIAL CONDITIONS
  • Job offer is contingent upon satisfactory clearance based on Background Check results.

  • Occasional evenings and weekends may be required.

  • Possession of, or ability to obtain, a favorable NACI adjudication is preferred but not required.

Pay Transparency Act

Annual Full Pay Range: Unclassified - No data available (will be prorated if the appointment percentage is less than 100%)

Hourly Equivalent: Unclassified - No data available

Factors in determining the appropriate compensation for a role include experience, skills, knowledge, abilities, education, licensure and certifications, and other business and organizational needs. The Hiring Pay Scale referenced in the job posting is the budgeted salary or hourly range that the University reasonably expects to pay for this position. The Annual Full Pay Range may be broader than what the University anticipates to pay for this position, based on internal equity, budget, and collective bargaining agreements (when applicable).

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Sr. High-Performance Computing Engineer - 141797
Sr. High-Performance Computing Engineer - 141797

University of California San Diego • United States

Hybrid
USD 130,000 - 164,000
HPC Systems Engineer - 132811
HPC Systems Engineer - 132811

UC San Diego/UC San Diego Health • United States

Hybrid
USD 94,000 - 115,000
HPC SYSTEMS ENGINEER
HPC SYSTEMS ENGINEER

UW Advancement Careers • Seattle (WA)

Hybrid
USD 97,000 - 158,000
HPC Systems Engineer - 132811
HPC Systems Engineer - 132811

UC San Diego • San Diego (CA)

Hybrid
USD 94,000 - 115,000
Cmptl and Data Sci Rsch Spec 4 - 142124
Cmptl and Data Sci Rsch Spec 4 - 142124

UC San Diego • San Diego (CA)

On-site
USD 60,000 - 90,000
Sr. HPC Systems Engineer (IT@JH Research Computing)
Sr. HPC Systems Engineer (IT@JH Research Computing)

John Hopkin’s University • Baltimore (MD)

On-site
USD 85,500 - 149,800
HPC SYSTEMS ENGINEER
HPC SYSTEMS ENGINEER

University of Washington • Seattle (WA)

Hybrid
USD 97,000 - 158,000
Cmptl and Data Sci Rsch Spec 4 - 142124
Cmptl and Data Sci Rsch Spec 4 - 142124

UC San Diego/UC San Diego Health • United States

Hybrid
USD 83,000 - 86,000
HPC Systems Engineer — Research Compute & Automation
HPC Systems Engineer — Research Compute & Automation

University of Washington • Seattle (WA)

Hybrid
USD 97,000 - 158,000
Systems and Network Administrator - 136522
Systems and Network Administrator - 136522

UC San Diego Health • San Diego (CA)

Hybrid
USD 96,862 - 143,947