AI Governance & Evaluation Engineer

Plutus21 Partners; Bogotá ciudad; On-site; COP 90,000,000 - 140,000,000; Full time

Job summary

Plutus21 is seeking an AI Engineer to join our core platform team and own agent evaluation and governance. You will build systems that test, approve, monitor AI agents, and enforce governance rules at runtime, working with the Principal Platform Engineer, Cloud Engineer, and product teams.

You’ll design evaluation workflows, capture data provenance, and ensure everything is auditable. This role focuses on production-grade governance for external client use.

Qualifications

• Strong software engineering skills in Python or TypeScript.
• Experience shipping multi-step LLM agents to production.
• Working knowledge of applied statistics (sample sizing, confidence bounds).
• Experience designing rule-based, in-code governance and approvals.

Responsibilities

• Build an evaluation framework with datasets, metrics, thresholds, and a go-live gate for agents.
• Run re-evaluations when models or prompts change and detect regressions.
• Create a governed agent runtime with ownership, approvals, and deterministic checks.
• Implement approval workflows for high-volume actions and auto-suspension when thresholds breached.
• Design review and evaluation workflows for AI-generated content sent outside the client org.
• Ensure all model calls are recorded with model, instruction version, data provenance, and checks.
• Keep platform independent of a single model provider.
• Translate governance rules into executable validation with other engineers and product teams.
• Document design decisions clearly before implementation.

Skills

Python

TypeScript

Software design

Testing

Documentation

Written communication

Tools

LinkML

SHACL

OWL/RDF

Job description

Plutus21 is seeking an AI Engineer to join our core platform team and own agent evaluation and governance. You will build systems that test, approve, monitor AI agents, and enforce governance rules at runtime, working with the Principal Platform Engineer, Cloud Engineer, and product teams.

You’ll design evaluation workflows, capture data provenance, and ensure everything is auditable. This role focuses on production-grade governance for external client use.