AI Infrastructure Engineer

Oho Group

San Francisco (CA)

On-site

USD 180,000 - 240,000

Full time

8 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Maintainer status on OSS project
Open-source contributions eligible for
GPU/ML compiler exposure

Job summary

Oho Group is seeking a Production Infrastructure Engineer to architect and own the agentic infrastructure layer of our platform. You will build gateways, runtimes, tool integrations, and security boundaries enabling autonomous agents to run safely in production.

You will deploy and scale runtimes, create MCP servers, and design zero-trust governance while driving observability and cost evaluations across the stack. Strong Linux, Kubernetes, TypeScript/Node.js, and Python skills are essential.

Qualifications

  • Experience designing and operating resilient backend/distributed systems.
  • Proficiency in Linux, containers, Kubernetes, and SSO.
  • Strong TypeScript/Node.js and Python for tooling and pipelines.
  • Public OSS contributions to agent frameworks or MCP projects.
  • Security-minded with least-privilege and sandboxing awareness.

Responsibilities

  • Deploy and scale agent runtimes across clusters and enterprise environments.
  • Build custom plugins and MCP servers to connect agents to CI/CD and scheduling systems.
  • Design zero-trust governance for tools, permissions, secrets, audits, and third-party plugins.
  • Develop telemetry for token usage, latency, cost per task, and regression evaluations.
  • Collaborate with runtime and compiler teams to translate workloads into system requirements.

Skills

Linux
Kubernetes
SSO
TypeScript/Node.js
Python

Tools

MCP Server/SDK
Open-source tooling
CI/CD
Container runtimes
Security/Zero-trust

Job description

We are working with a next-generation AI compute start-up building foundational technology for custom AI accelerators. With a proven engineering team and a working core platform already in place, they are entering a high-growth phase—transitioning from core development to full production delivery.

The Role

You will architect and own the agentic infrastructure layer of the platform—building the gateways, runtime orchestration, tool integrations, and security boundaries that allow autonomous agents to run safely and reliably in production.

What You’ll Do
  • Deploy & Scale Agent Runtimes: Harden and operate open-source agent runtimes and gateways across internal clusters and enterprise environments, integrating them with identity, networking, and observability stacks.
  • Build Core Tooling & MCP Integrations: Develop custom plugins and Model Context Protocol (MCP) servers that connect autonomous agents directly to engineering workflows, CI/CD pipelines, and cluster scheduling.
  • Architect Agent Security: Design zero-trust governance for tool-using agents, covering granular permissions, sandboxing, secrets management, audit logging, and supply-chain vetting for third-party plugins.
  • Drive Telemetry & Evaluations: Build observability around token usage, latency, and cost per task, alongside automated regression evaluations and failure analysis for upstream releases.
  • Collaborate Across the Stack: Partner with runtime and compiler teams to translate real-world agent workloads into system-level requirements, while contributing fixes upstream to keep internal forks minimal.
About you
  • Production Infrastructure Engineer: You have designed, deployed, and maintained backend or distributed systems that stayed resilient under real-world load, with strong command of Linux, containers, Kubernetes, and SSO.
  • TypeScript & Python Builder: You have deep proficiency in TypeScript and Node.js for building agent gateways and control planes, paired with solid Python for ML tooling and evaluation pipelines.
  • Open-Source Agent Contributor: You have a public record of meaningful contributions (merged PRs, issue triage, or RFC discussions) to open-source agent frameworks or MCP server/SDK repositories.
  • Systems & Security Mindset: You understand the internals of tool calling, memory management, and always-on execution, and you design proactively around least privilege and prompt injection risks.
  • AI-Native Workflow: You use AI-assisted coding tools daily to ship and debug faster, and you communicate clearly through actionable design docs.
Bonus
  • Maintainer or reviewer status on an open-source agent project, or authorship of a widely used MCP server or plugin.
  • Experience serving LLMs at scale (inference engines, batching, quantization, and accelerator scheduling) or operating clusters with mixed hardware.
  • Familiarity with GPU programming models or ML compiler stacks (e.g., LLVM, MLIR, Triton).
  • Background in red-teaming autonomous systems or building agent evaluation benchmarks.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Platform Infrastructure Engineer
Senior AI Platform Infrastructure Engineer

DXC Luxoft • United States

Remote
USD 150,000 - 190,000
AI Engineer - Agentic Systems
AI Engineer - Agentic Systems

vibehackers • United States

Remote
USD 120,000 - 180,000
AI Engineer
AI Engineer

STG • Salt Lake City (UT)

On-site
USD 120,000 - 170,000
AI Engineer
AI Engineer

Teserac • Santa Clara (CA)

On-site
USD 140,000 - 210,000
Health Care Plan
Paid Time Off
Free Food
+2
Senior AI Engineer, Release Engineering
Senior AI Engineer, Release Engineering

Intercontinental Exchange, Inc. • Atlanta (GA)

On-site
USD 140,000 - 190,000
AI Engineer - US
AI Engineer - US

ScaleOps • United States

On-site
USD 140,000 - 210,000
Agentic AI Developer
Agentic AI Developer

Xpetize • United States

Remote
USD 120,000 - 180,000
AI Engineer - Forward Deployed
AI Engineer - Forward Deployed

Moring, inc. • Atlanta (GA)

On-site
USD 120,000 - 160,000
Principal AI Engineer
Principal AI Engineer

Hearst • Dallas (TX)

On-site
USD 180,000 - 240,000
Lead Fullstack AI Engineer
Lead Fullstack AI Engineer

NetWeb Software • United States

Remote
USD 180,000 - 240,000