AI Infra Engineer: Scale AI Clusters on Kubernetes

Perplexity; San Francisco (CA); On-site; USD 243,000 - 405,000; Full time

Job summary

Perplexity is seeking an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS.

As an AI Infrastructure Engineer, you will partner closely with our Inference and Research teams to build, deploy, and optimize our large-scale AI training and inference clusters. You will design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads; manage Slurm HPC environments for distributed training;

Qualifications

• Experience deploying large Kubernetes deployments in production.
• Strong Slurm HPC workload management background.
• Distributed training systems and HPC environments experience.
• ML workloads require networking, storage, and compute management knowledge.
• Proficiency with PyTorch in distributed training contexts.

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads.
• Manage and optimize Slurm-based HPC environments for distributed training of large language models.
• Develop robust APIs and orchestration systems for training pipelines and inference services.
• Implement resource scheduling and job management across heterogeneous compute environments.
• Benchmark performance, diagnose bottlenecks, and improve both training and inference infrastructure.
• Build monitoring, alerting, and observability solutions for ML workloads on Kubernetes and Slurm.
• Respond to outages and collaborate across teams to maintain high uptime for training runs and inference services.
• Optimize cluster utilization and implement autoscaling for dynamic workloads.

Skills

Kubernetes admin

Slurm scheduling

Python

C++

PyTorch

LLM training

Tools

Terraform

Ansible

Job description

Perplexity is seeking an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS.

As an AI Infrastructure Engineer, you will partner closely with our Inference and Research teams to build, deploy, and optimize our large-scale AI training and inference clusters. You will design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads; manage Slurm HPC environments for distributed training;