Overnight HPC Operations Engineer – GPU, Slurm

Institute of Foundation Models; Sunnyvale (CA); On-site; USD 150,000 - 300,000; Full time

Benefits offered by this job

Comprehensive medical, dental, and vision benefits

Bonus

401K Plan

Generous paid time off

Paid Parental Leave

Employee Assistance Program

Life insurance and disability

Job summary

The Institute of Foundation Models in Sunnyvale, CA is hiring for a role focused on operational coverage during Abu Dhabi overnight hours. This position involves monitoring the health and performance of large-scale GPU clusters, responding to incidents, and supporting researchers with troubleshooting.

The ideal candidate will require a Bachelor's degree in a relevant field and at least 2 years of experience in Linux systems administration or related fields. Attractive benefits include comprehensive health coverage, bonuses, and a 401K Plan.

Qualifications

• 2+ years in Linux systems administration, SRE, DevOps, cloud operations, HPC, or infrastructure operations.
• Experience with scripting using Python or Bash.
• Exposure to AI/ML infrastructure and research computing environments.

Responsibilities

• Monitor health, performance, and availability of large-scale GPU clusters.
• Respond to incidents and perform first-level triage.
• Support researchers and troubleshoot job failures.
• Execute operational runbooks and recovery procedures.
• Validate cluster deployments, upgrades, and maintenance activities.
• Contribute to documentation and reporting.

Skills

Linux systems administration

Scripting (Python or Bash)

Strong Linux troubleshooting skills

Education

Bachelor's degree in a relevant field

Tools

Slurm

GPU infrastructure

AWS

Azure

GCP

Grafana

Prometheus

Datadog

Kubernetes

Job description

The Institute of Foundation Models in Sunnyvale, CA is hiring for a role focused on operational coverage during Abu Dhabi overnight hours. This position involves monitoring the health and performance of large-scale GPU clusters, responding to incidents, and supporting researchers with troubleshooting.

The ideal candidate will require a Bachelor's degree in a relevant field and at least 2 years of experience in Linux systems administration or related fields. Attractive benefits include comprehensive health coverage, bonuses, and a 401K Plan.