Senior AI QA Automation Engineer - Remote Impact & Quality

Encora; Comisión de Fomento de Perú; Remote; ARS 136,430,000 - 227,383,000; Full time

Job summary

Coforge is seeking a Senior AI QA Automation Engineer to design and implement automated evaluation frameworks for AI agents, focusing on AI quality, prompts, and RAG solutions. You will ensure end-to-end automation across UI, API, and integration layers, while monitoring production AI quality and implementing CI/CD gates.

Ideal candidates have 5+ years in software quality or reliability engineering, strong Python, and hands-on Playwright/Selenium experience, plus familiarity with Azure AI

Qualifications

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with experience building reusable automated testing frameworks.
• Hands-on experience with Playwright and/or Selenium.
• Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
• Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
• Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.

Responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Skills

Python

AI Testing

Test Automation

Playwright

Selenium

API Testing

Azure AI Foundry Evaluations

RAG Evaluation

Prompt Testing

CI/CD

Monitoring

QA Quality Concepts

Education

Bachelor’s degree in CS/Software Engineering/IS

Tools

Playwright

Selenium

CI/CD tooling

API testing tools

Job description

Coforge is seeking a Senior AI QA Automation Engineer to design and implement automated evaluation frameworks for AI agents, focusing on AI quality, prompts, and RAG solutions. You will ensure end-to-end automation across UI, API, and integration layers, while monitoring production AI quality and implementing CI/CD gates.

Ideal candidates have 5+ years in software quality or reliability engineering, strong Python, and hands-on Playwright/Selenium experience, plus familiarity with Azure AI