Senior Principal DevOps Engineer — AI Infra & Kubernetes

Upscale AI; United States; Hybrid; USD 263,000 - 284,000; Full time

Job summary

Upscale AI is building high‑performance AI infrastructure. We’re seeking an experienced SRE/DevOps leader to own reliability, deployment, and the platform infrastructure behind Orchestrator and AI Fabric environments.

You will operate Kubernetes clusters across on‑prem and cloud, design CI/CD pipelines, manage Terraform‑driven infrastructure, and handle secret rotations. You’ll also implement observability stacks and contribute to tooling that improves operational efficiency.

Qualifications

• 8–14 years in SRE, DevOps, or infrastructure engineering roles supporting production distributed systems.
• Deep Kubernetes expertise across cluster administration, networking, storage, RBAC, and troubleshooting in cloud and bare‑metal.
• Strong Terraform skills with multi‑environment, multi‑provider infrastructure at scale.
• Hands‑on experience building and operating CI/CD pipelines end‑to‑end.
• Production experience with observability tools: Prometheus, Grafana, Loki, Splunk, Datadog, or comparable.
• Scripting in Python, Bash, or Go; Linux systems internals and networking knowledge.
• Experience managing TLS/mTLS certificates and Vault or equivalents in production.
• Comfort working across AWS/GCP and on‑prem environments.

Responsibilities

• Own reliability, deployment, and operational infrastructure for Orchestrator and AI Fabric environments.
• Build and maintain Kubernetes clusters across on‑prem and cloud; design CI/CD pipelines with automated testing gates.
• Manage Terraform‑driven infrastructure with proper state management and drift detection.
• Operate the full observability stack (Prometheus, Grafana, Loki, Splunk, Datadog, Timestream) for end‑to‑end visibility.
• Automate secret and certificate rotations (TLS/MTLS, Vault) and credential lifecycle for multi‑tenant deployments.
• Lead incident response: runbooks, root cause analysis, post‑incident reviews with real fixes.
• Support customer deployments at site level; handle edge appliances and varying network constraints.
• Develop internal tooling and analytics to reduce toil and speed up engineering.
• Plan capacity, optimize costs, and forecast infrastructure needs as deployments scale.

Skills

Kubernetes

Terraform

CI/CD

Prometheus

Grafana

Loki

Datadog

Timestream

Splunk

Python

Bash

Go

Linux

TLS/MTLS

AWS

GCP

Troubleshooting

Tools

Rancher

Helm

GitHub Actions

Vault

Job description

Upscale AI is building high‑performance AI infrastructure. We’re seeking an experienced SRE/DevOps leader to own reliability, deployment, and the platform infrastructure behind Orchestrator and AI Fabric environments.

You will operate Kubernetes clusters across on‑prem and cloud, design CI/CD pipelines, manage Terraform‑driven infrastructure, and handle secret rotations. You’ll also implement observability stacks and contribute to tooling that improves operational efficiency.