Member of Technical Staff (AI Infrastructure Engineer)

Perplexity; San Francisco (CA); On-site; USD 243,000 - 405,000; Full time

Job summary

Perplexity is seeking an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS.

As an AI Infrastructure Engineer, you will partner closely with our Inference and Research teams to build, deploy, and optimize our large-scale AI training and inference clusters. You will design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads; manage Slurm HPC environments for distributed training;

Qualifications

• Experience deploying large Kubernetes deployments in production.
• Strong Slurm HPC workload management background.
• Distributed training systems and HPC environments experience.
• ML workloads require networking, storage, and compute management knowledge.
• Proficiency with PyTorch in distributed training contexts.

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads.
• Manage and optimize Slurm-based HPC environments for distributed training of large language models.
• Develop robust APIs and orchestration systems for training pipelines and inference services.
• Implement resource scheduling and job management across heterogeneous compute environments.
• Benchmark performance, diagnose bottlenecks, and improve both training and inference infrastructure.
• Build monitoring, alerting, and observability solutions for ML workloads on Kubernetes and Slurm.
• Respond to outages and collaborate across teams to maintain high uptime for training runs and inference services.
• Optimize cluster utilization and implement autoscaling for dynamic workloads.

Skills

Kubernetes admin

Slurm scheduling

Python

C++

PyTorch

LLM training

Tools

Terraform

Ansible

Job description

We are looking for an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS. As an AI Infrastructure Engineer, you will be partnering closely with our Inference and Research teams to build, deploy, and optimize our large-scale AI training and inference clusters.

Responsibilities

• Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads
• Manage and optimize Slurm-based HPC environments for distributed training of large language models
• Develop robust APIs and orchestration systems for both training pipelines and inference services
• Implement resource scheduling and job management systems across heterogeneous compute environments
• Benchmark system performance, diagnose bottlenecks, and implement improvements across both training and inference infrastructure
• Build monitoring, alerting, and observability solutions tailored to ML workloads running on Kubernetes and Slurm
• Respond swiftly to system outages and collaborate across teams to maintain high uptime for critical training runs and inference services
• Optimize cluster utilization and implement autoscaling strategies for dynamic workload demands

Qualifications

• Strong expertise in Kubernetes administration, including custom resource definitions, operators, and cluster management
• Hands‑on experience with Slurm workload management, including job scheduling, resource allocation, and cluster optimization
• Experience with deploying and managing distributed training systems at scale
• Deep understanding of container orchestration and distributed systems architecture
• High level familiarity with LLM architecture and training processes (Multi‑Head Attention, Multi/Grouped‑Query, distributed training strategies)
• Experience managing GPU clusters and optimizing compute resource utilization

Required Skills

• Expert‑level Kubernetes administration and YAML configuration management
• Proficiency with Slurm job scheduling, resource management, and cluster configuration
• Python and C++ programming with focus on systems and infrastructure automation
• Hands‑on experience with ML frameworks such as PyTorch in distributed training contexts
• Strong understanding of networking, storage, and compute resource management for ML workloads
• Experience developing APIs and managing distributed systems for both batch and real‑time workloads
• Solid debugging and monitoring skills with expertise in observability tools for containerized environments

Preferred Skills

• Experience with Kubernetes operators and custom controllers for ML workloads
• Advanced Slurm administration including multi‑cluster federation and advanced scheduling policies
• Familiarity with GPU cluster management and CUDA optimization
• Experience with other ML frameworks like TensorFlow or distributed training libraries
• Background in HPC environments, parallel computing, and high‑performance networking
• Knowledge of infrastructure as code (Terraform, Ansible) and GitOps practices
• Experience with container registries, image optimization, and multi‑stage builds for ML workloads

Required Experience

• Demonstrated experience managing large‑scale Kubernetes deployments in production environments
• Proven track record with Slurm cluster administration and HPC workload management
• Previous roles in SRE, DevOps, or Platform Engineering with focus on ML infrastructure
• Experience supporting both long‑running training jobs and high‑availability inference services
• Ideally, 3‑5 years of relevant experience in ML systems deployment with specific focus on cluster orchestration and resource management

Compensation Range: $220K - $405K