Senior AI QA Automation Engineer

Encora Inc.; Lima Metropolitana; À distance; PEN 206 000 - 413 000; Plein temps

Résumé du poste

Coforge is hiring a Senior AI QA Automation Engineer for remote work across Latin America. You will build and run automated evaluation frameworks for AI agents, focusing on prompts, RAG solutions, and model outputs, while validating AI behaviors such as hallucinations, grounding, and citation accuracy.

You will implement CI/CD gates, monitor production AI quality, and develop end-to-end automation for UI, API, integration, and UAT tasks.

Qualifications

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with experience building reusable automated testing frameworks.
• Hands-on experience with Playwright and/or Selenium.
• Experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
• Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
• Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
• Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
• Experience implementing CI/CD quality gates and production monitoring.
• Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.

Responsabilités

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, and prompt injection.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT.
• Implement CI/CD quality gates and monitor production AI quality, drift, reliability, and failure patterns.

Connaissances

Python programming

AI testing concepts

Test automation

Playwright

Selenium

API testing

CI/CD

Monitoring

UAT

Formation

Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or related field

Outils

Playwright

Selenium

API testing

CI/CD

Azure AI Foundry Evaluations

RAG evaluation

Prompt testing

Monitoring

Description du poste

Job Title: Senior AI QA Automation Engineer
Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring
Experience: 5+ YOE.
Location: Costa Rica, Peru, Colombia, Bolivia.
Mode: Remote
We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.

Key Responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Required Skills & Qualifications

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with experience building reusable automated testing frameworks.
• Hands-on experience with Playwright and/or Selenium.
• Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
• Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
• Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
• Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
• Experience implementing CI/CD quality gates and production monitoring.
• Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.
• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Skills

• Experience testing Generative AI, RAG solutions, copilots, or AI agents.
• Experience with LLM evaluation frameworks and advanced prompt testing.
• Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.
• Experience with adversarial testing and Responsible AI testing.
• Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.

At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.