On-Prem AI Inference DevOps Engineer - Hybrid/Remote

HellaGoodAudio; United States; Hybrid; USD 110,000 - 180,000; Full time

Benefits offered by this job

Flexible working hours

Hybrid or remote options depending on 

Private medical insurance

Life/Wellbeing benefits

Continuous learning

Job summary

HellaGoodAudio is seeking a DevOps Engineer to design, deploy, and maintain infrastructure for scalable AI inference workloads on-premises, ensuring secure, reliable deployments.

You will design GPU scheduling, model deployment pipelines, and data access patterns; build monitoring dashboards with Grafana, VictoriaMetrics, and runbooks; and partner with ML and platform teams to optimize performance of Kubernetes-based systems in a hybrid, on‑prem environment.

Qualifications

• Strong understanding of Kubernetes architecture and control plane components.
• Linux and networking troubleshooting skills for production Kubernetes clusters.
• Automation tooling using Python, Go, or Bash.
• Experience with IaC/configuration tools like Terraform or Ansible.

Responsibilities

• Design, develop, and maintain infrastructure for AI inference workloads in on-prem environments.
• Build and manage monitoring and observability tools with dashboards, alerts, and runbooks.
• Collaborate with ML engineers and platform teams to design scalable system architecture and test performance at scale.

Skills

Kubernetes architecture

Kubernetes operations

Linux networking troubleshooting

Automation with Python/Go/Bash

CI/CD workflows

Tools

Terraform

Ansible

VictoriaMetrics

Grafana

Argo CD

Job description

HellaGoodAudio is seeking a DevOps Engineer to design, deploy, and maintain infrastructure for scalable AI inference workloads on-premises, ensuring secure, reliable deployments.

You will design GPU scheduling, model deployment pipelines, and data access patterns; build monitoring dashboards with Grafana, VictoriaMetrics, and runbooks; and partner with ML and platform teams to optimize performance of Kubernetes-based systems in a hybrid, on‑prem environment.