AI QA Automation Engineer & Production Monitoring

Ron_Freelancer; Taguig; On-site; PHP 600,000 - 1,000,000; Full time

Job summary

Ron_Freelancer is seeking a QA engineer to design and execute testing strategies for AI-enabled workflows, ensuring accuracy, reliability, safety, and compliance through automated evaluation and monitoring.

You will build evaluation harnesses, runbooks, and dashboards using Playwright and Python, collaborate with developers on prompts and tool definitions, and ensure data privacy and regulatory alignment in production monitoring.

Qualifications

• Minimum 3 years’ experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)
• Strong Python skills for test automation, evaluation harnesses, and basic data analysis
• High attention to detail, with a focus on issues that materially impact reliability and user trust
• Comfort working with evolving tools, frameworks, and testing practices
• CI/CD pipelines and Git-based workflows

Responsibilities

• Contribution to auditability, risk management, and continuous quality improvement
• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
• Develop and maintain runbooks for common failure modes and contribute to incident response activities
• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

Skills

Python

Attention to detail

Team collaboration

Evidence-based decision making

Data analysis

Tools

Playwright

Deepeval

RAGAS

Evidently.AI

API testing

Job description

Ron_Freelancer is seeking a QA engineer to design and execute testing strategies for AI-enabled workflows, ensuring accuracy, reliability, safety, and compliance through automated evaluation and monitoring.

You will build evaluation harnesses, runbooks, and dashboards using Playwright and Python, collaborate with developers on prompts and tool definitions, and ensure data privacy and regulatory alignment in production monitoring.