Lead AI Inference Platform Engineer (Remote)

Karman, Inc.; Northern (KY); Hybrid; USD 170,000 - 210,000; Full time

Benefits offered by this job

Flexible PTO

Health benefits

401k match

Job summary

Karman, Inc. is seeking an AI Infrastructure Engineer to design and own the end-to-end AI inference platform across edge, cloud, and on-prem deployments. The role emphasizes reliability, performance, and scalability for production AI workloads.

You will work cross-functionally with product, engineering and data science teams, with remote work options and occasional on-site engagements. 5+ years in AI infra is preferred.

Qualifications

• 5+ years of software engineering focusing on AI infrastructure or distributed systems.
• Hands-on with AI model serving frameworks (e.g., vLLM, SGLang, Triton, TensorRT, TorchServe).
• Experience with container orchestration (Kubernetes, Docker).
• Experience deploying infra across datacenter/on-prem environments.
• Strong knowledge of GPU workloads and inference vs training.
• Proficient in Python; CUDA, C++, Go, Rust a plus.
• Excellent cross-functional communication; travel up to 10%.

Responsibilities

• Lead design and build of AI inference platform architecture and standards.
• Own end-to-end model serving infrastructure (on-prem and datacenter).
• Build fault-tolerant, high-performance systems for scalable AI inference.
• Collaborate with algorithms engineers to integrate inference data with power optimization.
• Optimize GPU utilization and inference performance across hardware fleet.
• Establish MLOps practices including CI/CD for deployment and monitoring.
• Contribute to roadmap decisions on build vs. buy and tooling.

Skills

AI infrastructure

Backend systems

Distributed systems

Python

C++

CUDA

Go

Rust

Kubernetes

Docker

Communication

Travel readiness

Tools

vLLM

SGLang

Triton

TensorRT

TorchServe

Terraform

Helm

Job description

Karman, Inc. is seeking an AI Infrastructure Engineer to design and own the end-to-end AI inference platform across edge, cloud, and on-prem deployments. The role emphasizes reliability, performance, and scalability for production AI workloads.

You will work cross-functionally with product, engineering and data science teams, with remote work options and occasional on-site engagements. 5+ years in AI infra is preferred.