AI QA Engineer: Automated Testing & Monitoring

Risewave Consulting Inc.; Taguig; On-site; PHP 600,000 - 1,200,000; Full time

Job summary

Risewave Consulting Inc. is seeking an Automation QA Engineer to define, implement, and continuously improve QA practices for AI-enabled solutions. The role focuses on accuracy, reliability, and safety, with automated evaluation and monitoring to maintain production readiness.

You will build automated evaluation harnesses, implement continuous monitoring, and collaborate with developers to refine prompts and workflows while ensuring compliance with data privacy standards.

Qualifications

• Minimum 3 years in QA, test automation, or DevOps roles (or 2 years with AI/ML testing exposure).
• Strong Python skills for test automation and data analysis.
• Attention to detail with focus on reliability and user trust.
• Experience with evolving tools and testing practices.
• Collaborative mindset applying evidence-based insights to product decisions.
• Proficiency in Python for automation, evaluation harnesses, and data analysis.
• Familiarity with UI automation using Playwright and API/workflow validation.
• Experience with AI evaluation frameworks and monitoring practices.

Responsibilities

• Define quality criteria and testing strategies for agent workflows, including accuracy and safety.
• Build automated evaluation harnesses to assess agent performance and compliance.
• Implement continuous production monitoring to detect anomalies and quality degradation.
• Develop and maintain automated test suites using Playwright and custom scripts.
• Apply LLM evaluation frameworks to assess output quality and drift.
• Produce dashboards and reports communicating quality metrics to engineering.
• Develop runbooks and contribute to incident response.
• Collaborate with developers to improve prompts, tool definitions, and workflows.
• Ensure testing and monitoring align with data privacy and regulatory requirements.

Skills

Python

QA automation

Playwright

DevOps

Data analysis

Monitoring

Anomaly detection

Collaboration

Scripting

Tools

Playwright

Deepeval

RAGAS

Evidently.AI

API scripting

Job description

Risewave Consulting Inc. is seeking an Automation QA Engineer to define, implement, and continuously improve QA practices for AI-enabled solutions. The role focuses on accuracy, reliability, and safety, with automated evaluation and monitoring to maintain production readiness.

You will build automated evaluation harnesses, implement continuous monitoring, and collaborate with developers to refine prompts and workflows while ensuring compliance with data privacy standards.