Inference Solutions Architect

The Edge Inference; City of Albany (NY); On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

Comprehensive benefits

Commission

Job summary

Edge Inference Inc. is building a nationwide AI inference network, a distributed fabric of Inference Points that delivers latency, throughput, and token-cost commitments customers rely on.

The Inference Solutions Architect converts a customer's model-serving workload into a concrete deployment: GPU platform, rack counts, serving-stack topology, KV-cache and batching strategy tailored to latency goals. You own the capacity math under every ICA — from MW reserved to GPU-hours committed — and

Qualifications

• 6+ years in solutions architecture or ML infrastructure.
• Experience serving LLMs in production with vLLM/TensorRT-LLM/Triton/SGLang.
• Ability to size GPU deployments from first principles (memory, KV-cache, batching).
• Knowledge of GPU infrastructure and scheduling (Kubernetes/Slurm).
• Experience owning capacity agreements, RFP responses, and architecture handoffs.

Responsibilities

• Translate customer workloads into Edge Inference deployments (model family, quantization, context length, concurrency).
• Select GPU platforms and defend choices on tokens per dollar.
• Define performance targets (TTFT, ITL, p50/p95/p99).
• Architect serving stack (vLLM, TensorRT-LLM, Triton, KServe, etc.).
• Design multi‑Inference Point placement for latency.
• Own capacity math for ICAs (MW, racks, GPU-hours).
• Build unit economics ($/million tokens, $/GPU-hour).
• Lead RFP responses and proofs of concept.
• Coordinate with infra to ensure power/thermal feasibility.
• Handoff deployments to Deployment/NIOC teams.

Skills

Solutions architecture

ML infrastructure

Customer-facing pre-sales

GPU platform engineering

Model deployment in production

TTFT and ITL tuning

Education

Bachelor's degree

Tools

vLLM

TensorRT-LLM

Triton

SGLang

KServe on Kubernetes

Ray Serve

NVIDIA Dynamo

Job description

Edge Inference Inc. is building the nationwide AI inference network — a distributed fabric of Inference Points that puts GPU compute within single-digit milliseconds of where inference is actually consumed. Customers do not buy space and power from us; they buy a place to serve models at a latency, a token throughput, and a cost per million tokens they cannot achieve from a distanthyperscale region. Every InferenceCapacity Agreement (ICA) we sign is, underneath the commercial terms, an engineering claim about tokens per second, time to first token, and GPU utilization. Someone has to make that claim credible before it is signed.

About the Role

The Inference Solutions Architect is the technical half of the ICA sales motion. You take a customer's model-serving workload — model family and parameter count, quantization, context length, concurrency, burst profile, latency target— and translate it into a concrete Edge Inference deployment: GPU platform selection, node and rack count, serving-stack topology, KV-cache and batching strategy, and multi-Inference Point placement that meets the customer's latency map. You own the capacity math that sits under the commercial proposal, and you are accountable for it being right. You partner with Enterprise Account Executives on Priority Class I–II pursuits, lead RFP technical response, design and run proofs of concept, and hand a validated architecture to the deployment and NIOC teams at signature.

Responsibilities

• Translate a customer's model-serving workload into a sized Edge Inference deployment — model family, parameter count, quantization (FP8/FP4/INT8), context length, concurrency, and burst profile mapped to GPU count, memory footprint, and rack count.
• Select the GPU platform per workload across NVIDIA GB200/GB300 NVL72, HGX H200, L40S, and RTX PRO Blackwell Server Edition, and AMD Instinct MI300X/MI355X, and defend the selection on tokens/sec per dollar rather than on brand.
• Set and validate performance targets with the customer: tokens per second, time to first token (TTFT) and inter-token latency (ITL) at p50/p95/p99, and the concurrency at which those targets must hold.
• Architect the serving stack for each engagement — vLLM, NVIDIA Dynamo, TensorRT-LLM, NIM microservices, Triton, SGLang, Ray Serve, KServe on Kubernetes — including disaggregated prefill/decode, speculative decoding, continuous batching, and KV-cache offload strategy.
• Design multi-Inference Point placement for latency: choose the IP set that puts the customer's users inside
• Own the capacity math behind every ICA — MW reserved, racks energized, GPU-hours committed, token-throughput commitments, and headroom for growth — and reconcile it with the deployment schedule so contracted capacity exists when it is promised.
• Build the unit-economics case with Finance: $/million tokens, $/GPU-hour, and revenue per MW energized, so the customer sees the cost argument and Edge Inference protects margin.
• Lead technical response on RFPs and RFIs, including architecture narrative, SLA commitments, security and sovereignty controls, and the compliance exhibits that Priority Class I–II customers require.
• Design, scope, and run proofs of concept on live Edge Inference capacity — define success criteria up front, benchmark honestly, and publish results that either win the deal or kill it early.
• Support power and thermal feasibility with the infrastructure engineering team: confirm that the proposed density fits the IP Standard Design envelope of 30–130 kW/rack with direct-to-chip liquid cooling, or flag the deviation before it is sold.
• Hand off signed deployments cleanly to the Deployment Project Management, provisioning, and NIOC teams with a documented architecture, SLA definitions, and monitoring requirements.
• Feed the field back into the product: bring recurring customer requirements — model formats, network patterns, compliance regimes, GPU platforms — into the IPSD and roadmap discussion.

Qualifications

• 6+ years in solutions architecture, ML infrastructure, or GPU platform engineering, with at least 2 years in a customer-facing pre-sales or field engineering role.
• Demonstrated hands-on experience serving large language models in production with vLLM, TensorRT-LLM, Triton, SGLang, or an equivalent stack — including measuring and tuning TTFT, ITL, and throughput.
• Ability to size GPU deployments from first principles: memory math, KV-cache growth with context and concurrency, batching behavior, and the throughput/latency tradeoff curve.
• Working knowledge of GPU infrastructure — NVLink/NVSwitch, InfiniBand or RoCEv2 east-west fabric, Kubernetes or Slurm scheduling, and GPU workload isolation between customers.
• Experience owning the technical content of large, multi-year infrastructure or capacity agreements, including SLA definition and RFP response.
• Credibility with both a customer's ML engineers and their executive sponsors — able to move between a benchmark discussion and a cost-per-token business case in the same meeting.
• Bachelor's degree in Computer Science, Electrical Engineering, or a related technical field, or equivalent production experience.
• Willingness to travel to customer sites and Inference Points as pursuits require.

Preferred Skills

• NVIDIA NCP-AIO or NCA-AIIO certification; CKA/CKAD.
• Experience at a neocloud,GPU cloud, or AI infrastructure provider selling capacityrather than software.
• Fine-tuning, distillation, or model-optimization experience that lets you advise customers on reducing serving cost, not just on buying more GPUs.
• Familiarity with sovereignty and regulated-workload requirements — data residency, FedRAMP-adjacent controls, or Priority Class I customer environments
• Experience with retrieval-augmented generation, agentic serving patterns, or multimodal inference workloads and their distinct latency profiles.
• Track record building reusable sizing tools, calculators, or reference architectures adopted by a sales organization.

Competitive base salary + commission and comprehensive benefits.