Remote Senior AI QA Automation Engineer

Encora10; Comisión de Fomento de Perú; Remoto; ARS 136.310.000 - 181.747.000; Tempo pieno

Descrizione del lavoro

Coforge is seeking a Senior AI QA Automation Engineer to design and implement automated evaluation frameworks for AI agents and RAG-powered workflows. You will build reusable testing infrastructure, run API/UI/integration end-to-end tests, and establish measurable quality thresholds for production AI systems.

The role emphasizes strong Python skills, Playwright or Selenium, CI/CD integration, and monitoring of AI quality, drift, and reliability across multi-region deployments.

Competenze

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with experience building reusable automated testing frameworks.
• Hands-on experience with Playwright and/or Selenium.
• Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
• Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
• Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
• Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Mansioni

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Conoscenze

Python

Test Automation

Playwright

Selenium

API Testing

CI/CD

AI Testing

Monitoring

Evaluation Metrics

UAT Automation

Formazione

Bachelor's degree in Computer Science or related field

Strumenti

Azure AI Foundry

RAG Evaluation

Descrizione del lavoro

Coforge is seeking a Senior AI QA Automation Engineer to design and implement automated evaluation frameworks for AI agents and RAG-powered workflows. You will build reusable testing infrastructure, run API/UI/integration end-to-end tests, and establish measurable quality thresholds for production AI systems.

The role emphasizes strong Python skills, Playwright or Selenium, CI/CD integration, and monitoring of AI quality, drift, and reliability across multi-region deployments.