AI Automation QA Engineer: Monitoring & Evaluation

IGO TECHONOLOGIES; Taguig; Hybrid; PHP 700.000 - 1.000.000; Vollzeit

Zusammenfassung

IGO TECHNOLOGIES is seeking an Automation QA Engineer to define and continuously improve QA practices for AI-enabled solutions. You will build automated evaluation harnesses, monitor production quality, and collaborate with developers to enhance prompts and workflows for reliability.

You will implement Playwright UI tests, API/workflow validation scripts, and maintain dashboards to communicate metrics to stakeholders while ensuring compliance with privacy and regulatory requirements.

Qualifikationen

• Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct AI/ML testing)
• Strong Python skills for test automation, evaluation harnesses, and basic data analysis
• High attention to detail affecting reliability and user trust
• Comfort with evolving tools, frameworks, and testing practices
• Collaborative mindset with evidence-based decision-making
• Programming: Python (test automation, evaluation harnesses, data analysis)
• UI Automation: Playwright (end-to-end testing)
• AI Evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, regression)
• Workflow Testing: API and agent workflow validation using custom scripts
• Monitoring: Production quality monitoring and anomaly detection

Aufgaben

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
• Develop and maintain runbooks for common failure modes and contribute to incident response activities
• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

Kenntnisse

Python

Test Automation

DevOps

Data Analysis

UI Automation

AI Evaluation

Collaboration

Tools

Playwright

Deepeval

RAGAS

Evidently.AI

Jobbeschreibung

IGO TECHNOLOGIES is seeking an Automation QA Engineer to define and continuously improve QA practices for AI-enabled solutions. You will build automated evaluation harnesses, monitor production quality, and collaborate with developers to enhance prompts and workflows for reliability.

You will implement Playwright UI tests, API/workflow validation scripts, and maintain dashboards to communicate metrics to stakeholders while ensuring compliance with privacy and regulatory requirements.