AI Infra Engineer: Scale ML Clusters on Kubernetes & Slurm

Perplexity; Palo Alto (CA); On-site; USD 220,000 - 405,000; Full time

Job summary

Perplexity is seeking an AI Infra engineer to join our growing team. You will design, deploy, and optimize large-scale AI training and inference clusters on AWS, using Kubernetes and Slurm to manage distributed workloads.

You will collaborate with Inference and Research teams to build robust APIs and orchestration systems, ensuring high availability and scalable performance across multi-tenant environments.

Qualifications

• Proven experience managing large-scale Kubernetes deployments in production.
• Hands-on Slurm HPC cluster administration and scheduling.
• Experience with ML frameworks in distributed training contexts (PyTorch).

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads.
• Manage and optimize Slurm-based HPC environments for distributed training of large language models.
• Develop robust APIs and orchestration systems for both training pipelines and inference services.
• Implement resource scheduling and job management systems across heterogeneous compute environments.
• Benchmark system performance, diagnose bottlenecks, and implement improvements across both training and inference infrastructure.
• Build monitoring, alerting, and observability solutions tailored to ML workloads running on Kubernetes and Slurm.
• Respond swiftly to system outages and collaborate across teams to maintain high uptime for critical training runs and inference services.
• Optimize cluster utilization and implement autoscaling strategies for dynamic workload demands.

Skills

Kubernetes administration

Slurm HPC

Python

C++

PyTorch

Distributed systems

LLM training

GPU resource management

Observability tools

APIs & orchestration

Tools

Kubernetes

Slurm

PyTorch

Terraform

Ansible

GitOps

CUDA

Job description

Perplexity is seeking an AI Infra engineer to join our growing team. You will design, deploy, and optimize large-scale AI training and inference clusters on AWS, using Kubernetes and Slurm to manage distributed workloads.

You will collaborate with Inference and Research teams to build robust APIs and orchestration systems, ensuring high availability and scalable performance across multi-tenant environments.