ML Inference Systems Engineer — Kubernetes & GPU Scale

Luma AI; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Luma AI seeks a systems engineer to own large-scale inference deployments. You will integrate new architectures into the inference engine, scale fleets across thousands of machines, and keep GPU pools busy while meeting SLOs.

This role emphasizes model serving, scheduling, and reliability at scale. Responsibilities include building tooling to measure and optimize inference workloads, collaborating across research and infra teams, and maintaining CI/CD for model checkpoints and SDKs.

Qualifications

• Strong Python and system-architecture skills.
• Experience deploying models with PyTorch, Hugging Face, vLLM, SGLang, TensorRT-LLM, or similar.
• Experience with queues, scheduling, traffic control, and fleet management at scale.
• Experience with Linux, Docker, and Kubernetes, and with orchestration, deployment, and scheduling.
• Familiarity with Redis and S3-compatible storage.
• Nice to Have: Modern networking stacks including RDMA (RoCE, InfiniBand, NVLink).
• High-performance large-scale ML systems (100+ GPUs).
• CUDA, and FFmpeg or multimedia processing.

Responsibilities

• Ship new model architectures by integrating them into the inference engine.
• Collaborate across research, engineering, and infrastructure to optimize model efficiency and deployments.
• Build internal tooling to measure, profile, and track the lifetime of inference jobs and workflows.
• Automate, test, and maintain inference services for maximum uptime and reliability.
• Manage and optimize inference workloads across clusters and hardware providers, and scale deployments across thousands of machines.
• Build scheduling systems that use expensive GPU resources optimally while meeting SLOs, and maintain CI/CD for model checkpoints and SDKs.

Skills

Python

System architecture

Model deployment

PyTorch

Hugging Face

vLLM

Scheduler & queues

Linux

Docker

Kubernetes

Redis

S3 storage

RDMA networking

CUDA

FFmpeg/multimedia

Tools

Docker

Kubernetes

Redis

S3-compatible storage

Job description

Luma AI seeks a systems engineer to own large-scale inference deployments. You will integrate new architectures into the inference engine, scale fleets across thousands of machines, and keep GPU pools busy while meeting SLOs.

This role emphasizes model serving, scheduling, and reliability at scale. Responsibilities include building tooling to measure and optimize inference workloads, collaborating across research and infra teams, and maintaining CI/CD for model checkpoints and SDKs.