Remote Senior AI QA Automation Engineer

Encora; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Coforge is actively seeking a Senior AI QA Automation Engineer to design and implement automated QA for AI-powered systems. You will create Python-based test frameworks, utilize Playwright/Selenium for UI testing, and validate APIs within CI/CD pipelines.

The role demands 5+ years in software quality engineering, a bachelor's degree, and expertise in AI evaluation, RAG, prompts, and monitoring production quality. Remote work across the Americas is supported.

Qualifications

• 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.
• Strong Python skills with experience building reusable automated testing frameworks.
• Hands-on experience with Playwright and/or Selenium.
• Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.
• Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.
• Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.
• Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.
• Experience implementing CI/CD quality gates and production monitoring.
• Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Responsibilities

• Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets and regression suites.
• Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.
• Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Skills

Python

Test Automation

AI quality concepts

Evaluation metrics

Education

Bachelor's degree in Computer Science or related field

Tools

Playwright

Selenium

API Testing

CI/CD

Azure AI Foundry Evaluations

RAG Evaluation

Prompt Testing

Job description

Coforge is actively seeking a Senior AI QA Automation Engineer to design and implement automated QA for AI-powered systems. You will create Python-based test frameworks, utilize Playwright/Selenium for UI testing, and validate APIs within CI/CD pipelines.

The role demands 5+ years in software quality engineering, a bachelor's degree, and expertise in AI evaluation, RAG, prompts, and monitoring production quality. Remote work across the Americas is supported.