Remote AI Infrastructure Engineer - Edge & Data Center

Karman; United States; Remote; USD 170,000 - 210,000; Full time

Job summary

Karman is a fast-growing NVIDIA-backed AI company enabling AI data centers to dynamically orchestrate power and unlock more compute capacity from existing energy infrastructure. The AI Infrastructure Engineer will design, build, and own the end-to-end infrastructure that serves AI models across edge, cloud, and data center integrations.

This role requires deep expertise in ML model serving, distributed systems, and GPU infrastructure, with cross-functional collaboration across product,

Qualifications

• 5+ years of software engineering experience with a strong focus on AI infrastructure, backend systems, or distributed systems.
• Hands-on experience with AI model serving frameworks (e.g., vLLM, SGLang, Triton, TensorRT, TorchServe, or similar)
• Understanding of container orchestration and cluster management (Kubernetes, Docker)
• Experience deploying and operating infrastructure across both datacenter and on-prem environments
• Strong knowledge of GPU workloads and the tradeoffs that come with them — you understand how inference differs from training, and why it matters
• Proficiency in Python; C++, CUDA, Go, Rust a plus
• Excellent communication skills and comfort working cross-functionally in a lean, fast-moving environment
• Willingness to travel up to 10% of time

Responsibilities

• Design and build the AI inference platform, establishing scalable architecture patterns, deployment standards, and ops practices.
• Own end-to-end model serving infrastructure for on-prem and datacenter deployments.
• Build fault-tolerant, high-performance AI model serving systems with low latency and cost efficiency.
• Collaborate with algorithms engineers to integrate inference data with power optimization.
• Optimize GPU utilization and inference performance across hardware, including NVIDIA accelerators.
• Establish MLOps practices: CI/CD for model deployment, monitoring, rollback.
• Contribute to infrastructure roadmap decisions and tooling selections.

Skills

AI infrastructure

Distributed systems

Python

C++

CUDA

Go

Rust

Kubernetes

Docker

Model serving frameworks

GPU workloads

Communication

Travel willingness

Tools

Kubernetes

Docker

Terraform

Helm

Datadog

Prometheus

Grafana

Job description

Karman is a fast-growing NVIDIA-backed AI company enabling AI data centers to dynamically orchestrate power and unlock more compute capacity from existing energy infrastructure. The AI Infrastructure Engineer will design, build, and own the end-to-end infrastructure that serves AI models across edge, cloud, and data center integrations.

This role requires deep expertise in ML model serving, distributed systems, and GPU infrastructure, with cross-functional collaboration across product,