Staff Engineer - Hosted AI Training Platform (Kubernetes)

Prime Intellect, Inc.; San Francisco (CA); Hybrid; USD 150,000 - 300,000; Full time

Benefits offered by this job

Remote or SF office flexibility

Visa sponsorship & relocation support

Professional development budget

Job summary

Prime Intellect, Inc. is building an open frontier AI platform. The role focuses on hosting training infrastructure and the Kubernetes-based orchestration that runs multi-cluster GPU workloads, with a strong emphasis on developer-facing surfaces and real-time monitoring.

You will design scalable training stacks, develop control-plane services in Python, and ship frontend UI components to support end-to-end model training and evaluation workflows in a hybrid work environment (remote or SF office).

Qualifications

• Strong Kubernetes operations experience across multi-cluster, multi-cloud environments.
• Experience building hosted training platforms and developer surfaces.
• Comfortable building FastAPI/REST/TaRPC services and frontend UIs.

Responsibilities

• Design and operate Kubernetes-based training and inference orchestration across multi-cluster, multi-cloud GPU fleets.
• Build and maintain Helm charts that compose trainers, inference servers, environment servers, and supporting services into reproducible Training stacks.
• Develop Python control-plane agents that watch pods and report run state to the platform.
• Implement scheduling and autoscaling for heterogeneous hardware using KEDA and gang scheduling.
• Run a GitOps workflow with PRs, Helm values, and CI.
• Build node-local model caches, checkpoint pipelines, and shared storage for fast cold starts.
• Operate observability stack (Prometheus, Grafana, Loki, DCGM) for GPU cluster debugging.

Skills

Kubernetes operations

Python backend development

Frontend development (TypeScript/React

Tools

Helm

Terraform

Prometheus

Grafana

DCGM

Job description

Prime Intellect, Inc. is building an open frontier AI platform. The role focuses on hosting training infrastructure and the Kubernetes-based orchestration that runs multi-cluster GPU workloads, with a strong emphasis on developer-facing surfaces and real-time monitoring.

You will design scalable training stacks, develop control-plane services in Python, and ship frontend UI components to support end-to-end model training and evaluation workflows in a hybrid work environment (remote or SF office).