AI QA Engineer: Automation, Monitoring & Evaluation

Quess Philippines Corp.; Taguig; Hybrid; PHP 700,000 - 1,200,000; Part time

Job summary

WTW seeks an Automation QA Engineer to define, implement, and continuously improve QA practices for AI-enabled and agent-based solutions. The role ensures systems meet standards for accuracy, reliability, performance, safety, and regulatory compliance, with automated evaluation and monitoring to support operational readiness.

You will build automated evaluation harnesses, implement production monitoring, and develop test suites using Playwright for UI and custom scripts for API/workflow

Qualifications

• Minimum 3 years in QA, test automation, or DevOps roles, or 2 years with direct testing AI/ML systems.
• Strong Python skills for test automation and data analysis.
• Attention to detail and reliability.
• Experience with evolving tools and testing practices.
• Collaborative, evidence-based approach to product decisions.
• Proficiency in Python for automation and data handling.
• UI Automation: Playwright for end-to-end testing.
• AI Evaluation: Deepeval, RAGAS, Evidently.AI for quality/drift analysis.
• Workflow testing: API and agent workflow validation with scripts.

Responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and risk.
• Build automated evaluation harnesses to assess agent performance, including hallucination rates and policy violations.
• Implement continuous production monitoring to detect anomalies and quality degradation.
• Develop and maintain automated test suites using Playwright for UI testing.
• Apply LLM evaluation frameworks to assess output quality and system drift over time.
• Produce dashboards and reports that communicate quality metrics to engineering and stakeholders.
• Develop runbooks for common failure modes and contribute to incident response.
• Collaborate with developers to improve prompts, tool definitions, and workflow designs based on results.
• Ensure testing, logging, and monitoring align with privacy, audit, and regulatory requirements.

Skills

Python

Playwright

Test automation

Data analysis

APIs

AI evaluation

Tools

Playwright

Deepeval

RAGAS

Evidently.AI

Job description

WTW seeks an Automation QA Engineer to define, implement, and continuously improve QA practices for AI-enabled and agent-based solutions. The role ensures systems meet standards for accuracy, reliability, performance, safety, and regulatory compliance, with automated evaluation and monitoring to support operational readiness.

You will build automated evaluation harnesses, implement production monitoring, and develop test suites using Playwright for UI and custom scripts for API/workflow