Agentic AI Application Tester

Willis Towers Watson; Manila; On-site; PHP 600,000 - 900,000; Full time

Job summary

Willis Towers Watson in Manila is seeking a QA engineer to define testing strategies for AI-enabled workflows, build evaluation harnesses, and monitor production systems.

The role emphasizes script-based validation, data privacy alignment, and collaboration with developers to improve prompts and tools, with a focus on reliability and operational risk.

Candidates should have strong Python, Playwright, and AI testing experience, and a detail-oriented mindset for dashboards and incident response.

Qualifications

• Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)
• Strong Python skills for test automation, evaluation harnesses, and basic data analysis
• High attention to detail, with a focus on issues that materially impact reliability and user trust
• Comfort working with evolving tools, frameworks, and testing practices
• Collaborative mindset, using evidence-based insights to influence product and engineering decisions

Responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
• Develop and maintain runbooks for common failure modes and contribute to incident response activities
• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

Skills

Python

Playwright

AI testing

Data analysis

QA automation

Education

Bachelor's degree in CS/Engineering

Tools

Playwright

Deepeval

RAGAS

Evidently.AI

API scripting

Job description

Key Accountabilities

• Definition and execution of testing and quality assurance strategies for AI-enabled workflows
• Continuous evaluation and monitoring of system behavior in production environments
• Contribution to auditability, risk management, and continuous quality improvement

Principal Responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
• Develop and maintain runbooks for common failure modes and contribute to incident response activities
• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

Knowledge, Skills & Experience

Essential

• Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)
• Strong Python skills for test automation, evaluation harnesses, and basic data analysis
• High attention to detail, with a focus on issues that materially impact reliability and user trust
• Comfort working with evolving tools, frameworks, and testing practices
• Collaborative mindset, using evidence-based insights to influence product and engineering decisions

Technical Skills (Required)

• Programming: Python (test automation, evaluation harnesses, data analysis)
• UI Automation: Playwright (end-to-end workflow testing)
• AI Evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, and regression analysis)
• Workflow Testing: API and agent workflow validation using custom scripts
• Monitoring: Production quality monitoring and anomaly detection

Desirable

• Pytest or equivalent testing frameworks
• SQL for querying logs, metrics, or evaluation datasets
• Prometheus, Grafana, or similar monitoring tools
• Familiarity with hallucination detection and AI safety patterns
• CI/CD pipelines and Git-based workflows

WTW is an Equal Opportunity Employer