Senior HPC Systems Engineer — GPU & Cluster Ops

UC San Diego; San Diego (CA); Sur place; USD 150 000 - 210 000; Plein temps

Résumé du poste

UC San Diego is seeking a Sr. HPC Engineer to lead the technical operation of the AWARE high-performance computing system.

You will manage compute and GPU nodes, networks, storage, and the job scheduler, collaborating with researchers and cyberinfrastructure leads to ensure reliability for near real-time forecasting and research workloads. The role involves planning future upgrades, mentoring staff, and maintaining thorough documentation while troubleshooting complex multi-layer systems across

Qualifications

• Advanced knowledge of systems integration deploying complex systems integration solutions.
• Advanced skills associated with system integration design, modification, implementation and deployment in a complex environment.
• Demonstrated ability to develop complex conversion and system implementation plans.
• Demonstrated ability to understand functional needs of the campus / medical center and / or UC system and how systems can support those needs.
• Demonstrated ability to interface with senior level technical and non-technical management on a regular basis.
• Demonstrated ability to create software specifications.
• Experience with identification and use of code libraries and open-source forums.
• Experience with planning for deployment and creation of feedback mechanisms.
• Experience developing and implementing complex project plans.
• Experience researching and evaluating new technology and solutions for complex environments.
• Ability to impart knowledge to senior level technical and non-technical audiences and to influence outcomes to meet objectives.
• Demonstrated ability to identify software and hardware interdependences in a complex multi-platform environment.
• Demonstrated ability to develop solutions for complex business and systems / custom / ad-hoc solutions.
• Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.
• Experience with complex multi-platform troubleshooting in a multi-platform environment.
• Demonstrated ability to plan and execute the technical aspects of deploying, upgrading, and decommissioning storage systems.
• Advanced knowledge and experience with monitoring tools, performance, and systems optimization, including intermediate knowledge of security for data, data transfers and systems.
• Experience with complex multi-platform troubleshooting in a multi-platform environment.
• Demonstrated knowledge of TCP/IP and Infiniband networking, including sockets, VLANs, and firewalls.
• Proven experience designing, deploying, and operating large-scale (petabyte-class) high-performance parallel and distributed file systems (e.g., Lustre, Ceph, BeeGFS, GPFS), as well as enterprise and local file systems (e.g., NFS, ZFS, ext4, XFS) in Linux-based environments, including troubleshooting and performance tuning.
• Expert knowledge and experience with large data storage arrays (more than100TB), and methods to administer, monitor and upgrade.
• Proven experience administering and supporting large-scale HPC clusters or other distributed POSIX (Linux) systems, including advanced knowledge of Linux system administration, primarily Red Hat and its derivatives (e.g., Rocky Linux).
• In depth experience in cluster management tasks including deployment, configuration, and troubleshooting of compute nodes, management nodes, network switches, and file servers.
• Demonstrated experience with scripting and automation using languages such as Bash and Python; use of configuration management tools (e.g., Ansible, CFEngine); and version control systems (e.g., Git) to manage and maintain system configurations and infrastructure.
• Advanced knowledge of HPC middleware stack including cluster management tools including Slurm, PBS, HPCM, and Bright Cluster Manager.
• Demonstrated experience evaluating new storage system technologies.

Responsabilités

• Lead the technical operation of the HPC system (AWARE) including administration, maintenance, upgrades, deployments, and troubleshooting.
• Ensure high availability for time-sensitive near realtime forecasting and research workloads; plan changes carefully to minimize disruption.
• Manage compute and GPU nodes, networks, parallel storage, job scheduling, authentication, monitoring and data-transfer services.
• Install and integrate new hardware/software; maintain configurations and automation; monitor performance and capacity; coordinate with campus teams and vendors.
• Troubleshoot jobs, MPI applications, software environments, containers, memory and I/O problems, and CPU/GPU performance across multiple layers.
• Assist in planning future additions/upgrades including gathering requirements, benchmarking, procurements, and deployment planning with minimal downtime.
• Share knowledge with staff; mentor, explain HPC concepts, review work, and improve operational documentation.
• Plan and execute deployment, upgrades, and decommissioning of storage systems; work with multi-platform environments.

Connaissances

Linux system administration

Slurm workload manager

BeegFS / Lustre storage

MPI & GPU computing

Python scripting

Automation (Ansible, -xcat)

Networking (TCP/IP, InfiniBand)

Performance tuning

Documentation & mentoring

Formation

Bachelor's degree in Computer Science, Engineering, or related field

Outils

Slurm

BeeGFS

Lustre

ZFS

GIT

Nagios / Prometheus / Telegraf

Description du poste

UC San Diego is seeking a Sr. HPC Engineer to lead the technical operation of the AWARE high-performance computing system.

You will manage compute and GPU nodes, networks, storage, and the job scheduler, collaborating with researchers and cyberinfrastructure leads to ensure reliability for near real-time forecasting and research workloads. The role involves planning future upgrades, mentoring staff, and maintaining thorough documentation while troubleshooting complex multi-layer systems across