Quality Engineer - AI Agents, Evals & Integrations

Cafeto Software USA, LLC; Bogotá; Presencial; COP 60.000.000 - 100.000.000; Jornada completa

Descripción de la vacante

Cafeto Software USA, LLC seeks a Senior Quality Engineer to strengthen the reliability and correctness of AI-powered billing and collections workflows, including AI agents, integrations, and event-driven processes.

You will define quality for AI-enabled workflows, design and build automated evaluations to detect regressions, and investigate failures across backend services, APIs, and cloud observability to ensure production readiness.

Formación

• Experience with quality engineering for AI-enabled systems and complex, high-consequence workflows.

Responsabilidades

• Define what “good” looks like for AI-enabled workflows and production quality.
• Build automated evaluations that detect meaningful regressions in AI agents, integrations, and workflows.
• Investigate failures across application behavior and cloud evidence to determine root causes.
• Establish confidence as changes move from refinement through production, including observability and monitoring for AI-powered processes.
• Focus on validating AI-agent behavior, financial and operational workflows, and integration points (backend services, APIs, event-driven processes).

Conocimientos

AI QA

AI non-determinism

Observability

Descripción del empleo

Overview

We are looking for a Senior Quality Engineer to help ensure the reliability, safety, and business correctness of AI-powered billing and collections workflows. This role focuses on AI agents, evaluations, integrations, backend services, APIs, event-driven workflows, and production observability. The person will help define what “good” looks like for AI-enabled workflows, build automated evaluations that detect meaningful regressions, investigate failures across application behavior and cloud evidence, and create confidence as changes move quickly from refinement through production. This is not primarily a browser automation or end-of-cycle manual testing role. Browser-based end-to-end coverage is important, but the core of this position is validating AI-agent behavior and the financial, operational, and integration workflows around it. The ideal candidate is comfortable working with imperfect requirements, non-deterministic AI behavior, external systems, asynchronous processing, and high-consequence workflows where incorrect outputs can affect invoices, collections activity, customer experience, or operational decisions.

Responsibilities

• Define what “good” looks like for AI-enabled workflows and production quality.
• Build automated evaluations that detect meaningful regressions in AI agents, integrations, and workflows.
• Investigate failures across application behavior and cloud evidence to determine root causes.
• Establish confidence as changes move from refinement through production, including observability and monitoring for AI-powered processes.
• Focus on validating AI-agent behavior, financial and operational workflows, and integration points (backend services, APIs, event-driven processes).

Qualifications

• Experience with quality engineering for AI-enabled systems and complex, high-consequence workflows.
• Ability to work with imperfect requirements and non-deterministic AI behavior.
• Familiarity with external systems, asynchronous processing, and production observability.
• Understanding that outputs can impact invoices, collections activity, customer experience, or operational decisions.