AI Architect

Insight Softmax Consulting; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Insight Softmax Consulting is hiring an AI Architect to lead discovery, design and delivery of AI infrastructure for customers building or operating AI platforms. You will prototype, validate assumptions, and hand designs to engineers for implementation.

The role requires hands-on experience with GPU-inference platforms, Kubernetes, and cloud/on-prem deployments. Remote work across multiple time zones is expected, with focus on international collaboration.

Qualifications

• At least 8 years in infrastructure, platform or ML engineering.
• 3+ years designing and operating production AI or ML platforms at scale.
• Hands-on deploying LLM inference in production (e.g., vLLM, TensorRT-LLM, llm-d, Triton).
• Deep knowledge of GPU infrastructure and GPU scheduling on Kubernetes.

Responsibilities

• Lead technical discovery with customers and translate goals into AI platform architectures.
• Design inference serving platforms on Kubernetes and bare metal including autoscaling and multi-tenancy.
• Define reference architectures for multi-tenant GPU clouds and on-prem deployments.
• Advise on AI governance: cost visibility, token accounting, data handling, and security posture for LLM services.
• Review implementations and own technical quality of outcomes.
• Represent the company in customer architecture reviews and pre-sales support.

Skills

Infrastructure engineering

ML platform design

Technical leadership

Tools

Kubernetes

Linux

GPU platforms

TensorRT-LLM

Triton

Envoy/Traefik

Job description

Insight Softmax is a leading organization in the fields of machine learning, data science, and high-performance computing (HPC). We are dedicated to providing innovative solutions in computing technology to solve complex problems in various sectors including research, science, manufacturing, finance, and data analytics. Description: We are hiring an AI Architect for our engineering team. You will be the senior technical voice for customers who are building or operating AI infrastructure: GPU clusters, inference platforms, AI gateways, and the services that sit on top of them. Our customers range from chip vendors and neoclouds running thousands of GPUs, to enterprises standing up their first production LLM services, to platform vendors who have us build parts of their product. They ask us how the leading AI labs and AI-native companies run their platforms, and they expect answers grounded in experience. You will lead discovery and architecture phases of engagements, produce reference architectures and recommendations, and hand clear designs to our engineers for implementation. You will also support pre-sales scoping and estimates. This is a hands-on architect role. You are expected to prototype, validate assumptions in a lab, and review the implementation, not only produce slides. Our team requires individuals with focus on delivery and results, hunger to learn, ability to adapt, and can thrive in a self-directed environment where you'll figure out what's needed to deliver rather than wait for detailed instructions.

Key Responsibilities

• Lead technical discovery with customers and translate business goals into AI platform architectures, phased roadmaps and effort estimates.
• Design inference serving platforms on Kubernetes and bare metal: model serving, GPU scheduling and partitioning, autoscaling, multi-tenancy, and capacity planning.
• Design AI gateway and routing layers: tenant-aware routing, model pools, budgets and quotas, rate limiting, observability, and multi-site failover.
• Define reference architectures for multi-tenant GPU clouds, sovereign and air-gapped deployments, and hybrid cloud and on-prem estates.
• Advise on AI usage governance: cost visibility, token accounting, audit, data handling, and security posture for LLM services.
• Evaluate and recommend build, buy or compose decisions across open source and commercial AI infrastructure components.
• Review implementations delivered by our AI and platform engineers and own the technical quality of the outcome.
• Represent Insight Softmax in customer architecture reviews, partner enablement sessions and executive readouts.
• Reserve time each week for pre-sales support, internal platform work and mentoring.

Qualifications

• At least 8 years in infrastructure, platform or ML engineering, with at least 3 years designing and operating production AI or ML platforms at scale.
• Hands-on experience deploying LLM inference in production (for example vLLM, TensorRT-LLM, llm-d, Triton) including tensor and pipeline parallelism, batching, caching and performance tuning.
• Deep understanding of GPU infrastructure: NVIDIA and AMD accelerators, drivers and runtimes, GPU scheduling on Kubernetes, high-speed interconnects and storage for AI workloads.
• Experience designing AI gateway or API management layers (for example LiteLLM, Traefik, Envoy or Tetrate based designs, Kubernetes Gateway API inference extensions).
• Strong Kubernetes and Linux skills.
• Able to read and debug the stack from the scheduler down to the node.
• Deep knowledge of at least one of AWS, GCP, or Azure.
• Preferably AWS.
• Track record of leading architecture engagements with enterprise customers, including writing proposals, estimates and architecture documents that engineers can execute against.
• Clear written and spoken communication with both engineers and executives.

Optional Experience and Skills

• Experience at an AI lab, AI-native company, or neocloud, or building platform products for them.
• Multi-tenant GPU cloud design, including namespace and cluster-level isolation, quota, billing and chargeback.
• Slurm or other batch schedulers alongside Kubernetes for mixed HPC and AI estates.
• RAG, agentic systems, model evaluation and fine-tuning pipelines.
• Air-gapped, FIPS or other regulated deployment experience.
• Spectro Cloud Palette and PaletteAI, Run:ai, or similar GPU platform products.
• Observability for AI services: traces, token and cost metrics, quality monitoring.

Bonus Experience and Skills

• Python, Go or Rust.
• Software engineering background beyond infrastructure.
• Published talks, blog posts or open source contributions in AI infrastructure.
• Using linux as your primary desktop workstation environment.

Attributes that we value

• Transparency
• Grit
• Effectiveness
• Willingness to learn

Location and Hours: Full time position Remote work environment 80% of your work schedule will be during the business time zones for Latin America and Canada. Some customer meetings will occur each week at non-standard / nighttime ho