AI Quality Engineer: End-to-End AI Eval & CI/CD Reliability

Cafeto Software; Mexico; On-site; PHP 5,474,000 - 7,908,000; Full time

Job summary

Cafeto Software is seeking a senior QE/AI Quality engineer to own the evaluation strategy for AI-powered workflows, including real billing and collections outcomes, agent behavior, output quality, and business-rule compliance.

You will build automated evaluations (DeepEval and similar), integrate them into CI/CD, test end-to-end workflows, and use Langfuse, CloudWatch, and traces to distinguish product defects from model behavior. Strong collaboration with product and engineering is required.

Qualifications

• 5+ years in QE/SDET or AI quality roles.
• Hands-on testing of LLM/AI/RAG/agentic systems.
• Experience building AI evaluations (DeepEval preferred; others count).
• AI observability tooling: Langfuse, LangSmith, Arize Phoenix, or equivalent.
• Strong AWS + CloudWatch - logs, metrics, traces, alarms.
• API testing (pytest, Postman, Bruno, or equivalent) and SQL data validation.
• Experience with async/event-driven systems and CI/CD integration.

Responsibilities

• Own the evaluation strategy for AI-powered workflows (billing/collections outcomes).
• Build automated AI evaluations combining deterministic checks and LLM-based judgments.
• Integrate evaluations into CI/CD so AI regressions are caught before release.
• Test full end-to-end workflows: APIs, back-end services, async processing, third‑party integrations.
• Investigate failures using Langfuse, CloudWatch, logs, and traces to distinguish defects from model behavior.

Skills

AI QA

SDET

LLM testing

AI observability

AWS CloudWatch

API testing

SQL validation

CI/CD integration

Async systems

Tools

DeepEval

Ragas

Promptfoo

LangSmith

TruLens

Arize

Langfuse

Arize Phoenix

Job description

Cafeto Software is seeking a senior QE/AI Quality engineer to own the evaluation strategy for AI-powered workflows, including real billing and collections outcomes, agent behavior, output quality, and business-rule compliance.

You will build automated evaluations (DeepEval and similar), integrate them into CI/CD, test end-to-end workflows, and use Langfuse, CloudWatch, and traces to distinguish product defects from model behavior. Strong collaboration with product and engineering is required.