AI Infra Engineer: Scale ML Clusters with Kubernetes

Perplexity; California (MO); On-site; USD 140,000 - 190,000; Full time

Job summary

Perplexity is seeking an AI Infra engineer to design, deploy, and optimize scalable Kubernetes clusters and Slurm HPC environments for AI training and inference on AWS. You will partner with the Inference and Research teams to build robust pipelines and schedulers for large-scale models.

Ideal candidates have hands-on Kubernetes and Slurm expertise, experience with distributed training systems, and proficiency in Python and C++.

Qualifications

• Expert-level Kubernetes administration and YAML configuration management.
• Hands-on experience with Slurm workload management and HPC environments.
• Experience deploying and managing distributed training systems at scale.

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads.
• Manage and optimize Slurm-based HPC environments for distributed training of large language models.
• Develop robust APIs and orchestration systems for training pipelines and inference services.
• Implement resource scheduling and job management across heterogeneous compute environments.
• Benchmark system performance and diagnose bottlenecks in training and inference infra.
• Build monitoring, alerting, and observability for ML workloads on Kubernetes and Slurm.
• Respond to outages and collaborate across teams to maintain high uptime for training runs and inference services.
• Optimize cluster utilization and implement autoscaling strategies for dynamic workloads.

Skills

Kubernetes admin

Slurm scheduling

Python

C++

PyTorch

Distributed systems

Networking

Observability

Tools

Terraform

Ansible

Job description

Perplexity is seeking an AI Infra engineer to design, deploy, and optimize scalable Kubernetes clusters and Slurm HPC environments for AI training and inference on AWS. You will partner with the Inference and Research teams to build robust pipelines and schedulers for large-scale models.

Ideal candidates have hands-on Kubernetes and Slurm expertise, experience with distributed training systems, and proficiency in Python and C++.