AI Governance Engineer: Evaluation & Safe Release

Plutus21; Puerto Bogotá; Sur place; COP 70 000 000 - 110 000 000; Plein temps

Résumé du poste

Plutus21 is seeking an AI Engineer to join our core platform team and own agent evaluation and governance. You will build systems to test, approve, and monitor AI agents with runtime governance, ensuring rules are enforced in code and that datastores and provenance are tracked.

You will collaborate with platform, cloud, and client teams to define capabilities, datasets, and quality thresholds, while keeping the platform model-provider agnostic and scalable.

Qualifications

• Proven experience shipping tool-using, multi-step LLM agents to production environments, beyond prototypes.
• Demonstrated experience building evaluation sets and metrics, setting thresholds, and catching regressions across model or prompt changes.
• Working knowledge of applied statistics, including sample sizing, confidence bounds, and estimating error rates from samples.
• Experience designing systems in which rules such as approvals, checks, and kill switches are enforced in code rather than left to the model.
• Strong software engineering skills in Python or TypeScript, including testing, code review, and building maintainable services.
• Strong written communication skills, with the ability to produce clear design notes and work from documented decisions.
• Experience with knowledge graphs or ontologies (such as LinkML, SHACL, or OWL/RDF) and translating domain rules into validation is a plus.
• Experience building human-in-the-loop systems, including review queues, escalation paths, and approval workflows, is a plus.
• Experience with retrieval over governed data, including access control and data classification, is a plus.

Responsabilités

• Build and maintain an evaluation framework covering datasets, metrics, agreed thresholds, and a go-live gate for each agent capability.
• Run re-evaluations whenever an agent's model or instructions change, and identify regressions before release.
• Build a governed agent runtime that assigns an accountable owner to each action, requires human approval where needed, and applies deterministic checks before any irreversible action.
• Implement rule-level and campaign-level approval for high-volume actions, including statistically sized review samples before release and automatic suspension when quality thresholds are breached.
• Design review and evaluation workflows for AI-generated content that is sent outside the client organization.
• Ensure every model call is recorded with the model used, instruction version, data provenance, and quality checks applied.
• Keep the platform independent of any single model provider.
• Work with the Principal Platform Engineer to convert written governance rules into executable validation.
• Collaborate with product owners and client engagement teams to define capability requirements, evaluation datasets, and quality thresholds.
• Document design decisions clearly before implementation begins.

Connaissances

LLM agents

Evaluation metrics

Python

TypeScript

Statistics

Governance rules

Code-level approvals

Knowledge graphs

Retrieval systems

Multi-model governance

Outils

LinkML

SHACL

OWL/RDF

Description du poste

Plutus21 is seeking an AI Engineer to join our core platform team and own agent evaluation and governance. You will build systems to test, approve, and monitor AI agents with runtime governance, ensuring rules are enforced in code and that datastores and provenance are tracked.

You will collaborate with platform, cloud, and client teams to define capabilities, datasets, and quality thresholds, while keeping the platform model-provider agnostic and scalable.