HPC Lead System Administrator

KAUST (King Abdullah University of Science and Technology); Thuwal; On-site; SAR 300,000 - 450,000; Full time

Job summary

KAUST in Thuwal, Saudi Arabia, seeks a Lead HPC Systems Engineer to oversee a 300+ GPU/CPU Linux cluster, parallel filesystems, and high-performance networks.

You will supervise 3–4 HPC system administrators, develop SOPs, implement workload policies, and drive advanced research infrastructure while collaborating with faculty, staff, and students.

Qualifications

• Bachelor's degree in a relevant discipline with 10+ years of experience.
• Master's degree in a relevant discipline with 7+ years of experience.
• PhD in a relevant discipline with 5+ years of experience.

Responsibilities

• Ensure smooth operation of the Linux cluster (300+ nodes) and high-performance networks.
• Develop and enforce SOPs for the system and the team.
• Supervise 3–4 HPC system administrators.
• Plan upgrades and workload scheduler policy implementations.
• Maintain and support high-performance filesystems and TCP/IP/HPC networks.
• Mentor staff and collaborate with faculty, staff and students.
• Lead staffing and project planning for major HPC initiatives.
• Engage in problem solving and innovative approaches to improve infrastructure.

Skills

Team leadership

Systems administration

Scripting Bash/Python/Ruby

Networking (Infiniband)

Project management

Education

Bachelor of Science in a relevant discipline

Master of Science in a relevant discipline

Doctor of Philosophy in a relevant discipline

Tools

Puppet

Job description

Serve as the Lead for the team ensuring smooth operation of the Linux cluster consisting of 300+ GPU/CPU compute nodes including parallel filesystems and high-performance network. This is partly technical and partly people leading role which involves supervision of 3-4 experienced HPC system administrators. The role involves development, implementation and supervision of standard operating procedures for the system and the team.

Major Responsibilities

• System operation and upgrade planning to meet laboratory and customer requirements
• Workload scheduler policy development and implementation
• Support of high-performance filesystems
• Network infrastructure management including TCP/IP and HPC networks
• Use of scripting languages for nodes automation and configuration management
• Hardware failures and spare part management
• Build effective relationships with staff, faculty and students through the Core Labs
• Manages multiple or significant projects which may require the use of sophisticated project planning techniques
• Plans, schedules, conducts, or coordinates detailed phases of the work of a major project or in a total project of moderate scope
• Identifies technical training needs for staff attached to the area
• Serve as a resource and as a member to respond to security and safety incidents
• Creates opportunities to enhance technical methodology or content through expansion of existing, or development of, new efforts; may extend technology into new application areas; contributes or leads in major intellectual development activities
• Provides innovative problem-solving approaches to enhance organizational capabilities; uses peer network to expand technical capabilities and identify new research opportunities
• Understands broad strategic objectives and contributes to them; nurtures and maintains relationships with major customers
• May initiate new project concepts; develops technical proposals and makes presentations to potential customers
• Will supervise several scientists, engineers or technicians on assigned work; provides major input to staffing of overall project teams; builds teams and staff to optimize efficiency and cost effectiveness
• Identifies and evaluates candidates for open positions; mentors/trains staff in development of technical, project and business development skills
• TCP/IP and high performance networks (Infiniband)
• Proficient in scripting languages (i.e. Bash, Python, Ruby)
• Familiar with configuration management tools (Puppet)
• Will have working level contact with users and suppliers
• Demonstrates an analytical and systematic approach to problem solving
• Takes the initiative in identifying and negotiating appropriate development opportunities
• Demonstrates effective communication skills in written and oral English
• Works effectively with other teams in the Supercomputing Laboratory
• Plans, schedules and monitors own work (and that of others) competently within limited deadlines and according to relevant legislation and procedures
• Ability to work successfully in a highly collaborative research environment
• Uses discretion in identifying and resolving complex problems and assignments
• Performs a broad range of work, sometimes complex and non-routine, in a variety of environments
• Maintain expert-level knowledge in most of the laboratory systems, including high performance computing systems administration, high performance storage administration, or high performance network administration

Qualifications and Experience

• Bachelor of Science (or equivalent) in a relevant discipline plus 10 years’ experience, OR Master of Science (or equivalent) in a relevant discipline plus 7 years’ experience OR Doctor of Philosophy (or equivalent) in a relevant discipline plus 5 years’ experience.