Principal AI Inference Platform Engineer - GPU/K8s

Lila Sciences; Cambridge (ME); On-site; USD 192,000 - 272,000; Full time

Benefits offered by this job

Medical, dental, and vision

Life and disability insurance

Flexible time off

Parental leave

Educational assistance

Commuter benefits

Lunch program

Job summary

Lila Sciences is seeking a Staff/Principal DevOps Engineer - AI Inference to design, implement, and optimize infrastructure for scalable ML inference. You will bridge platform engineering, SRE, and ML infra to power low-latency, high-throughput inference across GPU clusters and cloud accelerators.

You will collaborate with ML engineers, researchers, and software engineers to build reliable inference platforms that serve production users while maximizing compute efficiency.

Qualifications

• Expertise in DevOps, SRE, or Platform Engineering for GPU/accelerator infra at scale.
• Deep experience with Kubernetes for ML workloads: GPU scheduling, quotas, node affinity, accelerator management.
• Strong Python proficiency for automation and ML tooling.
• Hands-on deploying to AWS with Terraform/Helm and GPU compute (EKS, EC2 P-series/Inf/Trn).
• Experience with model serving infra: inference servers, batching, KV-cache, LLM serving frameworks.

Responsibilities

• Design, implement, and optimize GPU-accelerator infra for ML inference at scale.
• Build inference platforms with low latency and high throughput across GPU clusters and cloud accelerators.
• Develop multi-tenant, topology-aware scheduling and resource isolation for GPUs.
• Create production-grade deployment pipelines with canary rollouts, A/B testing, and safe rollbacks across regions.
• Implement observability: GPU utilization, latency profiling, token throughput dashboards, SLO/SLI tracking.
• Drive infrastructure-as-code with Terraform and Helm for EKS clusters and accelerator networking.

Skills

DevOps

SRE

Kubernetes

Python

AWS

LLM serving

GPU scheduling

Tools

Terraform

Helm

CUDA

Driver dependencies

Triton Inference Server

vLLM

Job description

Lila Sciences is seeking a Staff/Principal DevOps Engineer - AI Inference to design, implement, and optimize infrastructure for scalable ML inference. You will bridge platform engineering, SRE, and ML infra to power low-latency, high-throughput inference across GPU clusters and cloud accelerators.

You will collaborate with ML engineers, researchers, and software engineers to build reliable inference platforms that serve production users while maximizing compute efficiency.