AI Operations & QA Engineer - Hybrid

E-team Workforce Private Corporation; Taguig; On-site; PHP 700,000 - 1,100,000; Full time

Job summary

E-team Workforce Private Corporation is seeking an Automation QA Engineer to define and improve QA practices for AI-enabled solutions. You will build automated evaluation harnesses, implement production monitoring, and create test suites using Playwright for UI, API, and workflow validation.

You will apply AI evaluation frameworks to measure quality, drift, and regression over time, and contribute to incident response with runbooks and dashboards that communicate metrics to stakeholders.

Qualifications

• Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)
• Strong Python skills for test automation, evaluation Harnesses, and basic data analysis
• High attention to detail, focusing on issues impacting reliability and trust
• Comfort with evolving tools, frameworks, and testing practices
• Collaborative mindset, using evidence-based insights to influence product decisions
• Programming: Python (test automation, evaluation harnesses, data analysis)
• UI automation: Playwright (end-to-end testing)
• AI evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, regression)
• Workflow testing: API and agent workflow validation using custom scripts
• Monitoring: Production quality monitoring and anomaly detection

Responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk
• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success
• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns
• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation
• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time
• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders
• Develop and maintain runbooks for common failure modes and contribute to incident response activities
• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results
• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

Skills

Python

Playwright

QA automation

Data analysis

CI/CD

SQL

AI evaluation

Tools

Deepeval

RAGAS

Evidently.AI

Playwright

Job description

About the role

The Automation QA Engineer defines, implements, and continuously improves quality assurance practices for AI-enabled and agent-based solutions. The role ensures systems meet agreed standards for accuracy, reliability, performance, safety, and compliance, and supports operational readiness through automated evaluation and monitoring.

Key responsibilities

• Define quality criteria and testing strategies for agent workflows, covering accuracy, latency, safety, compliance, and operational risk


• Build automated evaluation harnesses to assess agent performance, including hallucination rates, tool misuse, policy violations, and task success


• Implement continuous production monitoring to detect anomalies, quality degradation, and emerging safety concerns


• Develop and maintain automated test suites using Playwright for UI testing and custom scripts for API and workflow validation


• Apply LLM evaluation frameworks to assess output quality, regression, and system drift over time


• Produce and maintain dashboards and reports that communicate quality metrics and trends to engineering and stakeholders


• Develop and maintain runbooks for common failure modes and contribute to incident response activities


• Collaborate closely with developers to improve prompts, tool definitions, and workflow designs based on test results


• Ensure testing, logging, and monitoring practices align with data privacy, audit, and regulatory requirements

About you

• Minimum 3 years' experience in QA, test automation, or DevOps roles (or 2 years with direct experience testing AI or ML-enabled systems)


• Strong Python skills for test automation, evaluation harnesses, and basic data analysis


• High attention to detail, with a focus on issues that materially impact reliability and user trust


• Comfort working with evolving tools, frameworks, and testing practices


• Collaborative mindset, using evidence-based insights to influence product and engineering decisions


• Programming: Python (test automation, evaluation harnesses, data analysis)


• UI automation: Playwright (end-to-end workflow testing)


• AI evaluation: Deepeval, RAGAS, Evidently.AI (LLM quality, drift, and regression analysis)


• Workflow testing: API and agent workflow validation using custom scripts


• Monitoring: Production quality monitoring and anomaly detection

Preferred skills/experience

• Pytest or equivalent testing frameworks


• SQL for querying logs, metrics, or evaluation datasets


• Prometheus, Grafana, or similar monitoring tools


• Familiarity with hallucination detection and AI safety patterns


• CI/CD pipelines and Git-based workflows