Remote Senior AI QA Automation Engineer (AI Testing & CI/CD)

Encora Inc.; Bogotá ciudad; Remote; COP 286,615,000 - 414,000,000; Full time

Job summary

Coforge is hiring a Senior AI QA Automation Engineer to lead the creation of automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, and model outputs. You will define evaluation metrics, benchmark datasets, regression suites, and release thresholds to assure quality.

You will test AI-specific behaviors such as hallucinations, grounding, citation accuracy, and prompt injection, while building UI, API, integration, performance, and end-to-end automation.

Qualifications

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with experience building reusable automated testing frameworks.
• Hands-on experience with Playwright and/or Selenium.
• Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
• Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
• Experience defining evaluation metrics, benchmarks, regression suites, and measurable quality thresholds.
• Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
• Experience implementing CI/CD quality gates and production monitoring.
• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Skills

Python

AI Testing

Test Automation

Playwright

Selenium

API Testing

Azure AI Foundry Evaluations

RAG Evaluation

Prompt Testing

CI/CD

Monitoring

Education

Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field

Job description

Coforge is hiring a Senior AI QA Automation Engineer to lead the creation of automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, and model outputs. You will define evaluation metrics, benchmark datasets, regression suites, and release thresholds to assure quality.

You will test AI-specific behaviors such as hallucinations, grounding, citation accuracy, and prompt injection, while building UI, API, integration, performance, and end-to-end automation.