HPC Observability Engineer

EIT Professionals Corp; United States; Remote; USD 100,000 - 130,000; Full time

Job summary

EIT Professionals Corp is seeking a skilled HPC Observability Engineer to work remotely. The ideal candidate will have extensive experience in setting up Grafana dashboards and maintaining InfluxDB services. You will be responsible for developing Python scripts for data ingestion and visualizing critical metrics in a high-performance computing environment. This role requires strong analytical skills and the ability to work independently, with a focus on high-quality code and documentation.

Qualifications

• 5-8 years of experience with Grafana, InfluxDB, and Telegraf.
• Knowledge of Docker and Google Cloud Platform is advantageous.
• Experience in Python and Bash scripting is a plus.

Responsibilities

• Develop and leverage Grafana dashboards and Telegraf configurations.
• Visualize non-native resources in Grafana.
• Collaborate with teams to troubleshoot and optimize pipelines.

Skills

Python

Communication

Analysis

Documentation

Education

B.Tech, MS, or PhD in Computer Science

Tools

Grafana

InfluxDB

Telegraf

Docker

Git

Google Cloud Platform

Job description

22 hours ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

Direct message the job poster from EIT Professionals Corp

Role: HPC Observability Engineer (Python, HPC)

Location: Remote

Contract

Description:

The client has Grafana and InfluxDB services running on K8S in-house on-premises. Telegraf is used to ingest data from a GPU HPC cluster into InfluxDB. This engineer will help collect and visualize data for the “Terra” platform. The HPC Observability Engineer should have experience in:

• Setting up and maintaining Grafana dashboards for HPC environments
• Creating drill-down dashboards for servers, including metrics like memory, network, and CPU utilization
• Exploring and utilizing out-of-the-box metrics from InfluxDB
• Writing Python scripts for data ingestion into InfluxDB with examples
• Developing a proof of concept with a simple Python script to monitor load
• Ingesting Infiniband packet data
• Monitoring LSF jobs in various states
• Visualizing server-specific and cluster-wide metrics in Grafana
• Optional: Integrating third-party plugins like DDN’s Lustre, Mellanox fabric, etc.

Qualifications and Skills:

• B.Tech, MS, or PhD in Computer Science or related field
• 5-8 years of experience with Grafana, InfluxDB, and Telegraf
• Experience in Python and Bash scripting is a plus
• Knowledge of Docker and Google Cloud Platform is advantageous
• HPC operations experience is beneficial
• Strong communication skills and ability to work independently
• Proficiency in requirements analysis and automated testing
• Ability to write efficient, secure, and well-documented Python code
• Experience with Git and pipeline development
• Awareness of modern security and development practices

Responsibilities:

• Develop and leverage Grafana dashboards and Telegraf configurations
• Create dashboards for server and cluster metrics
• Develop Python scripts for data ingestion and documentation
• Visualize non-native resources in Grafana
• Optional: Integrate third-party plugins
• Maintain high-quality code and documentation
• Collaborate with teams to troubleshoot and optimize pipelines

Desired Skills:

• Python (good to have)
• Bash scripting (good to have)
• Docker (must)
• HPC operations and LSF (good to have)
• Experience with DDN Lustre, Mellanox fabric (good to have)
• Google Cloud Platform (good to have)
• Knowledge of Git (must)

Seniority level:

• Mid-Senior level

Employment type:

• Contract

Job function:

• Engineering and Information Technology

Industries:

• IT Services and IT Consulting

This job is active and accepting applications.