Remote Senior AI QA Automation Engineer

Encora10; Colombia; Remote; COP 286,469,000 - 381,959,000; Full time

Job summary

Coforge is hiring a Senior AI QA Automation Engineer for remote work across multiple regions including Costa Rica, Peru, Colombia and others. You will build automated evaluation frameworks for AI agents and RAG solutions, covering benchmarks, golden responses, and regression suites.

The role emphasizes testing AI behaviors, prompt testing, and multi-agent coordination. Ideal candidates have 5+ years in QA automation, strong Python, and hands-on Playwright/Selenium experience, with proficiency in

Qualifications

• 5+ years in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with reusable automated testing frameworks.
• Hands-on with Playwright and/or Selenium.
• Experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.

Responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, plus UI, API, integration, performance, and end-to-end automation.
• Implement CI/CD quality gates and monitor production AI quality, drift, reliability, and recurring failure patterns.

Skills

Python

AI Testing

Test Automation

Playwright

Selenium

API Testing

Azure AI Foundry

RAG Evaluation

Prompt Testing

CI/CD

Monitoring

Education

Bachelor’s degree in CS/SE

Tools

Playwright

Selenium

CI/CD pipelines

API testing tools

Job description

Coforge is hiring a Senior AI QA Automation Engineer for remote work across multiple regions including Costa Rica, Peru, Colombia and others. You will build automated evaluation frameworks for AI agents and RAG solutions, covering benchmarks, golden responses, and regression suites.

The role emphasizes testing AI behaviors, prompt testing, and multi-agent coordination. Ideal candidates have 5+ years in QA automation, strong Python, and hands-on Playwright/Selenium experience, with proficiency in