AI OS Engineer

Harvey Nash; Dallas (TX); Remote; USD 120,000 - 180,000; Full time

Job summary

Harvey Nash is seeking an experienced AI OS Engineer for a high-impact, 6-month contract to lead the architecture and integration of an AI Operating System—an orchestration framework for autonomous agents, multi-LLM routing, memory, tool execution, and local-to-cloud compute pipelines.

You will turn architectural blueprints into production-grade infrastructure, develop real-time evaluation frameworks, and optimize latency and compute costs in a remote US-based setting with potential extension.

Qualifications

• 5+ years of production software engineering experience, with 2+ years focused on building agentic frameworks, multi-agent orchestrations, or LLM infrastructure.
• Advanced proficiency in Python, TypeScript/Node.js, and modern async execution models.
• Hands-on expertise with agent architectures and orchestration frameworks (e.g., LangGraph, AutoGen, CrewAI, LlamaIndex, or custom in-house runtimes).
• Proven track record working with vector databases, embedding systems, and hybrid RAG implementations.
• Direct experience with Docker, Kubernetes, vLLM / Triton inference engines, and cloud platforms (AWS Sagemaker, GCP Vertex AI, or Azure ML).
• Mastery of RESTful/gRPC APIs, message queues (Kafka, RabbitMQ, Redis), and microservice architectures.

Responsibilities

• Design, build, and deploy agentic workflows, dynamic task schedulers, and execution runtime environments powering internal AI applications.
• Implement robust retrieval systems, long-term state persistence, vector databases (e.g., pgvector, Qdrant, Pinecone), and hybrid-search mechanisms to optimize agent context windows.
• Architect multi-model routing layers (e.g., Anthropic, OpenAI, open-source foundation models) for cost-efficiency, fallback management, and low-latency inference.
• Develop secure sandbox environments for tool execution, code generation, API calls, and agent safety protocols.
• Build evaluation harnesses to track model drift, execution accuracy, hallucination rates, and latency bottlenecks.
• Containerize and deploy AI OS infrastructure on cloud environments (AWS / GCP / Azure) using CI/CD pipelines.

Skills

Python

TypeScript/Node.js

Async programming

LLM infra concepts

RESTful APIs / gRPC

Tools

Docker

Kubernetes

vLLM/Triton

AWS GCP Azure

Job description

AI Operating System (AI OS) Engineer - Contract

Position Type: 6-Month Contract (W2 / C2C)

Location: Remote (US-Based)

Duration: 6 Months (Potential for Extension)

Position Overview

We are seeking an experienced AI OS Engineer for a high-impact, 6-month contract initiative. In this role, you will lead the architecture and integration of our next-generation AI Operating System (AI OS)—a core orchestration framework designed to seamlessly manage autonomous agents, multi-LLM routing, context memory systems, tool execution, and local-to-cloud compute pipelines.

Because this is a 6-month deliverable-driven contract, you will focus on turning architectural blueprints into production-grade infrastructure, executing real-time evaluation frameworks, and optimizing latency and compute costs.

Key Responsibilities

• Design, build, and deploy agentic workflows, dynamic task schedulers, and execution runtime environments powering internal AI applications.

• Implement robust retrieval systems, long-term state persistence, vector databases (e.g., pgvector, Qdrant, Pinecone), and hybrid-search mechanisms to optimize agent context windows.

• Architect multi-model routing layers (e.g., Anthropic, OpenAI, open-source foundation models) for cost-efficiency, fallback management, and low-latency inference.

• Develop secure sandbox environments for tool execution, code generation, API calls, and agent safety protocols.

• Build evaluation harnesses to track model drift, execution accuracy, hallucination rates, and latency bottlenecks.

• Containerize and deploy AI OS infrastructure on cloud environments (AWS / GCP / Azure) using CI/CD pipelines.

Required Qualifications

• 5+ years of production software engineering experience, with 2+ years focused on building agentic frameworks, multi-agent orchestrations, or LLM infrastructure.

• Advanced proficiency in Python, TypeScript/Node.js, and modern async execution models.

• Hands-on expertise with agent architectures and orchestration frameworks (e.g., LangGraph, AutoGen, CrewAI, LlamaIndex, or custom in-house runtimes).

• Proven track record working with vector databases, embedding systems, and hybrid RAG implementations.

• Direct experience with Docker, Kubernetes, vLLM / Triton inference engines, and cloud platforms (AWS Sagemaker, GCP Vertex AI, or Azure ML).

• Mastery of RESTful/gRPC APIs, message queues (Kafka, RabbitMQ, Redis), and microservice architectures.

Preferred Qualifications

• Experience with local LLM serving, quantization methods (AWQ, GGUF), and self-hosted foundation models (Llama, Mistral).

• Deep understanding of sandboxed execution environments (e.g., WebAssembly, Docker-in-Docker, E2B) for safe AI agent tool execution.

• Prior contract experience operating in fast-paced, 6-month delivery cycles with clear milestone check-ins.

Contract Milestones & Deliverables

• Finalize system architecture, set up local/cloud runtime execution environments, and deploy the core orchestration layer.

• Integrate multi-agent tool execution, long-term memory state persistence, and guardrail protocols.

• Conduct system-wide evaluation harness benchmarking, latency/cost optimization, and handoff documentation for internal engineering teams.