AI Quality Assurance & Automation Engineer

Willis Towers Watson; Pateros; On-site; PHP 700,000 - 1,100,000; Full time

Job summary

Willis Towers Watson in Metro Manila is seeking a QA engineer to define testing strategies for AI-enabled workflows and to build automated evaluation harnesses. You will monitor production behavior, develop dashboards, and contribute to incident response while ensuring privacy and regulatory compliance.

The role requires strong Python skills, Playwright experience, and collaboration with developers to improve prompts and tool definitions. On-site position based in the Philippines.

Qualifications

• Minimum 3 years’ experience in QA, test automation, or DevOps roles (or 2 years with direct AI/ML testing experience).
• Strong Python skills for test automation, evaluation harnesses, and basic data analysis.
• High attention to detail with focus on reliability and user trust.
• Comfort adapting to evolving tools, frameworks, and testing practices.
• Collaborative mindset using evidence-based insights to influence product decisions.

Responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk.
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success.
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns.
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation.
• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time.
• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders.
• Develop and maintain runbooks for common failure modes and contribute to incident response activities.
• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results.
• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements.

Skills

QA experience

Test automation

DevOps

Python

Attention to detail

Collaboration

Tools

Playwright

Deepeval

RAGAS

Evidently.AI

API validation scripts

Prometheus

Grafana

Job description

Willis Towers Watson in Metro Manila is seeking a QA engineer to define testing strategies for AI-enabled workflows and to build automated evaluation harnesses. You will monitor production behavior, develop dashboards, and contribute to incident response while ensuring privacy and regulatory compliance.

The role requires strong Python skills, Playwright experience, and collaboration with developers to improve prompts and tool definitions. On-site position based in the Philippines.