AI Agent Orchestration Engineer and Architect

A2B2.ai; Hong Kong; Remote; HKD 900,000 - 1,300,000; Full time

Job summary

A2B2.ai is hiring a senior backend engineer to design and operate the orchestration layer for production multi-agent workflows. You will own how agents choose next steps, exchange state, call tools, recover from failure, request human approval and emit traces for explainability.

You will work across Python services, workflow state, retrieval, tool contracts, evaluation and cloud deployment. Collaborating with product, data and domain teams, you’ll ensure engineering quality of the runtime layer.

Qualifications

• 3+ years building backend or AI/ML systems in production.
• Deep Python proficiency with FastAPI or a comparable framework.
• Hands-on with LangGraph, LangChain, CrewAI, AutoGen or equivalent orchestration.
• Experience managing workflow state, retries, persistence and tool calls across APIs.
• Practical experience with LLM APIs (OpenAI, Claude, Gemini) and vector search.
• Strong system-design judgment, Git and code-review discipline, testing habits and clear communication.

Responsibilities

• Design supervisor-worker, planner-executor, hierarchical and specialist-agent workflows.
• Define agent responsibilities, inputs/outputs, state transitions and failure paths.
• Build long-running workflows with checkpoints, resumability, retries and idempotency.
• Set the boundary between deterministic code paths and model-led decisions.
• Integrate agents with databases, APIs and enterprise systems.
• Design safe tool contracts and validation around external actions.
• Build memory/context systems while avoiding data leakage.
• Work with PostgreSQL, vector search and asynchronous processing for scalable state.
• Instrument traces, telemetry and cost/latency/accuracy metrics across workflow stages.
• Develop production-ready engineering docs, runbooks and CI/CD processes.
• Maintain non-advisory boundary in code with evidence and uncertainty in outputs.

Skills

Python proficiency

FastAPI

Async patterns

LangGraph/LangChain/CrewAI

API error handling

System design

Git and code review

LLM APIs and vector search

Distributed/backend systems

Clear written communication

Tools

PostgreSQL

Redis

Docker

Kubernetes

CI/CD

Cloud (AWS/GCP/Azure)

Job description

Platform

A2B2.ai is the intelligence platform for the future of wealth.

Reporting line

Technology

Location

Hong Kong (we will consider other locations / remote for right profile and experience).

Company Description

A2B2 is a hybrid intelligence and service platform for individual investors and wealth professionals, combining machine reach with human judgement. Individual investors use A2B2's agentic personal, private, and family offices (Arcus, Bridge, Cypher) to stay on top of their holdings, prepare for conversations with their own bankers and advisors, and compound their wealth with more confidence over time. Wealth professionals - IFAs, EAMs, family offices, asset managers, insurance agents, and bank relationship managers - use A2B2 Decoder to compare products, prepare for client meetings, and manage their practice, so they can nurture existing relationships and grow their client base. CIOs and enterprises also use A2B2 to extend their reach and influence. A2B2 doesn't sell financial products, offer investment advice, or take commissions, keeping its answers free of any agenda but the user's own.

Role Description

You will design and operate the orchestration layer for production multi-agent workflows. You will own how agents choose next steps, exchange state, call tools, recover from failure, request human approval and emit the traces needed to explain what happened.

This is a senior hands-on engineering role across Python services, workflow state, retrieval, tool contracts, evaluation and cloud deployment. You will work with product, data and domain colleagues while remaining accountable for the engineering quality of the runtime layer.

What you will do

Multi-agent architechure

• Design supervisor-worker, planner-executor, hierarchical and specialist-agent workflows.
• Define agent responsibilities, inputs, outputs, state transitions, failure paths and completion criteria.
• Build long-running workflows with checkpoints, resumability, retries, idempotency and durable state.
• Set the boundary between deterministic code paths and model-led decisions, using deterministic controls where the risk or task demands them.

Tool calling, state and memory

• Integrate agents with databases, APIs, enterprise systems and the Model Context Protocol (MCP).
• Design safe tool contracts, structured outputs, permission boundaries and validation around external actions.
• Build memory and context systems that retain useful state without confusing, leaking or fabricating prior information.
• Work with PostgreSQL, vector search and asynchronous processing to keep retrieval and agent state predictable at scale.

Safety, observability and evaluation

• Implement human approval steps, policy filters, confidence thresholds, escalation paths and action-level audit trails.
• Instrument traces, telemetry and cost, latency and accuracy metrics across each workflow stage.
• Build evaluation harnesses that test agent decisions, tool selection, retrieval quality and failure handling before release and during operation.
• Investigate production incidents from traces and data, then improve the system rather than only patching the symptom.

Production Engineering

• Build Python services and APIs, including async processing, rate limits, caching, graceful degradation and test coverage.
• Containerise and deploy workloads using Docker, Kubernetes and cloud services such as AWS, GCP or Azure AI Foundry.
• Contribute to CI/CD, code review, architecture decisions, runbooks and engineering documentation.
• Maintain the non-advisory boundary in code, ensuring outputs present evidence and uncertainty rather than personalised investment instructions.

What you bring

• 3+ years building and operating backend, distributed or AI/ML systems in a production setting.
• Deep Python proficiency, including FastAPI or a comparable backend framework, async patterns, APIs and error handling.
• Hands-on delivery with LangGraph, LangChain, CrewAI, AutoGen, native agent SDKs or an equivalent orchestration approach. You can explain the agent decision loop, not only name the framework.
• Experience managing workflow state, retries, persistence and tool or function calling across APIs, databases or enterprise systems.
• Practical experience with LLM APIs such as OpenAI, Claude, Gemini or open-source models, plus RAG, embeddings and vector search.
• Working knowledge of PostgreSQL and data modelling. Redis, pgvector or similar components are useful.
• Experience with tracing, logging, evaluation or test harnesses that measure agent and LLM-system behaviour.
• Experience shipping containerised services with Docker, Kubernetes, CI/CD and cloud infrastructure.
• Strong system-design judgment, Git and code-review discipline, testing habits and clear written communication.

Nice to have

• Model Context Protocol (MCP), Agent-to-Agent (A2A) or comparable tool-communication protocols.
• Durable workflow engines such as Temporal, or event-driven and distributed workflow execution.
• Human-in-the-loop control design for consequential actions, including approval queues and policy enforcement.
• LLM evaluation methods, prompt optimisation, fine-tuning or operation of local and open-source models.
• Financial, regulated, data-intensive or high-trust product experience.
• TypeScript or Go for service boundaries, performance-sensitive components or platform tooling.

What strong evidence looks like

• We will look for a system you can explain end to end. Strong evidence includes a clear workflow graph, explicit state design, tool-call validation, human approvals where needed, traces from real runs, an evaluation method and a production deployment story.
• A retrieve-then-answer RAG application, a model router without autonomous decisions, or coding-assistant use alone does not demonstrate agent orchestration depth for this role.