AI Engineer

Plutus21 Partners; Bogotá ciudad; On-site; COP 90,000,000 - 140,000,000; Full time

Job summary

Plutus21 is seeking an AI Engineer to join our core platform team and own agent evaluation and governance. You will build systems that test, approve, monitor AI agents, and enforce governance rules at runtime, working with the Principal Platform Engineer, Cloud Engineer, and product teams.

You’ll design evaluation workflows, capture data provenance, and ensure everything is auditable. This role focuses on production-grade governance for external client use.

Qualifications

• Strong software engineering skills in Python or TypeScript.
• Experience shipping multi-step LLM agents to production.
• Working knowledge of applied statistics (sample sizing, confidence bounds).
• Experience designing rule-based, in-code governance and approvals.

Responsibilities

• Build an evaluation framework with datasets, metrics, thresholds, and a go-live gate for agents.
• Run re-evaluations when models or prompts change and detect regressions.
• Create a governed agent runtime with ownership, approvals, and deterministic checks.
• Implement approval workflows for high-volume actions and auto-suspension when thresholds breached.
• Design review and evaluation workflows for AI-generated content sent outside the client org.
• Ensure all model calls are recorded with model, instruction version, data provenance, and checks.
• Keep platform independent of a single model provider.
• Translate governance rules into executable validation with other engineers and product teams.
• Document design decisions clearly before implementation.

Skills

Python

TypeScript

Software design

Testing

Documentation

Written communication

Tools

LinkML

SHACL

OWL/RDF

Job description

Job Summary

Plutus21 is seeking an AI Engineer to join our core platform team and take ownership of agent evaluation and governance. The AI Engineer will build the systems that test, approve, and monitor AI agents before and after they go live, and will ensure that governance rules are enforced in code at runtime. This role works closely with the Principal Platform Engineer, the Cloud Engineer, and product and client engagement teams, and is critical to preparing AI capabilities for use by external clients.

Key Responsibilities

• Build and maintain an evaluation framework covering datasets, metrics, agreed thresholds, and a go-live gate for each agent capability.
• Run re-evaluations whenever an agent's model or instructions change, and identify regressions before release.
• Build a governed agent runtime that assigns an accountable owner to each action, requires human approval where needed, and applies deterministic checks before any irreversible action.
• Implement rule-level and campaign-level approval for high-volume actions, including statistically sized review samples before release and automatic suspension when quality thresholds are breached.
• Design review and evaluation workflows for AI-generated content that is sent outside the client organization.
• Ensure every model call is recorded with the model used, instruction version, data provenance, and quality checks applied.
• Keep the platform independent of any single model provider.
• Work with the Principal Platform Engineer to convert written governance rules into executable validation.
• Collaborate with product owners and client engagement teams to define capability requirements, evaluation datasets, and quality thresholds.
• Document design decisions clearly before implementation begins.

Qualifications

• Proven experience shipping tool-using, multi-step LLM agents to production environments, beyond prototypes.
• Demonstrated experience building evaluation sets and metrics, setting thresholds, and catching regressions across model or prompt changes.
• Working knowledge of applied statistics, including sample sizing, confidence bounds, and estimating error rates from samples.
• Experience designing systems in which rules such as approvals, checks, and kill switches are enforced in code rather than left to the model.
• Strong software engineering skills in Python or TypeScript, including testing, code review, and building maintainable services.
• Strong written communication skills, with the ability to produce clear design notes and work from documented decisions.
• Experience with knowledge graphs or ontologies (such as LinkML, SHACL, or OWL/RDF) and translating domain rules into validation is a plus.
• Experience building human-in-the-loop systems, including review queues, escalation paths, and approval workflows, is a plus.
• Experience with retrieval over governed data, including access control and data classification, is a plus.
• Experience in regulated domains such as financial services or privacy-sensitive environments is a plus.
• Experience working with more than one model provider is a plus.