AI Platform Engineer – Agent Orchestration & Evaluation

Bonnie; Amsterdam; Hybrid; EUR 70,000 - 110,000; Full time

Benefits offered by this job

25 holiday days

Meaningful equity

New office on the Herengracht

Full setup: MacBook and tools budget

Office in Amsterdam

Job summary

Bonnie is hiring engineers to own the decision layer, tool-sets, and the evaluation system for a growing AI-assisted phone and chat service. You will design orchestration across models, craft a schema-driven configuration platform, and ensure reliability across millions of conversations in a hybrid Amsterdam setting.

The stack includes Python, FastAPI, LiteLLM, LangFuse, Langsmith, MySQL/Postgres, Docker, and modern AI tools.

Qualifications

• 3+ years building and operating production backends.
• Agentic systems, and where they break — tool calling, multi-step orchestration, retries, context management.
• A point of view on evaluation — proving AI got better, not just a demo.
• Async Python and API design — contracts, timeouts, idempotency, failure modes.
• AI-native with accountability — coding agents in daily work.
• Comfort in a codebase you did not write — large system used by 500+ customers.
• Amsterdam area and EU work authorization — no visa sponsorship.

Responsibilities

• Design and own Bonnie's tool-sets — availability, reservations, modifications, escalation with contracts and latency budgets.
• Build the evaluation system — replay conversations, score outcomes, catch regressions before shipping.
• Build the configuration platform — schema-driven model built from blocks for per-business behavior, versioned and safe to change without deploy.
• Orchestrate the models — routing, fallbacks, guardrails, cost and latency trade-offs.
• Work on the call path — latency budgets, transfers, SIP, load balancing across STT and TTS providers.
• Run what you build — deploys, observability, and a two-week support shift roughly quarterly.

Skills

Backend development

LLM systems

Async Python

API design

Relational databases

Tools

Python

FastAPI

LiteLLM

LangFuse

Langsmith

MySQL

PostgreSQL

Docker

Claude Code

Codex

Cursor

Job description

Bonnie is hiring engineers to own the decision layer, tool-sets, and the evaluation system for a growing AI-assisted phone and chat service. You will design orchestration across models, craft a schema-driven configuration platform, and ensure reliability across millions of conversations in a hybrid Amsterdam setting.

The stack includes Python, FastAPI, LiteLLM, LangFuse, Langsmith, MySQL/Postgres, Docker, and modern AI tools.