AI QA Engineer: Automated Testing for AI Apps

ConnectOS; Metro Manila; Hybrid; PHP 670,000 - 1,116,000; Full time

Benefits offered by this job

Hybrid Work Arrangement

Medical, Dental Coverage and Life ins.

Paid Vacation and Sick Leave (Quarter

Competitive salary package and annual

Financial Assistance Program

13th Month Pay

Complimentary Office Amenities

Regular Company Events

Metro Manila location

Job summary

ConnectOS is seeking an Automation Engineer to own QA for AI-enabled features within a Data and AI team. The role focuses on automated testing, AI service quality, agent workflow validation, and multi-tenant security across a distributed UK and Philippines pattern.

You will design and maintain automated test coverage for AI-enabled applications, APIs, and backend services, and work on agent-driven user journeys with strong ownership and proactive risk management.

Qualifications

• Strong QA fundamentals across manual, exploratory and automated testing.
• Hands-on automation experience using Playwright, Selenium, Cypress or similar frameworks.
• API testing experience using Postman, REST Assured or equivalent tooling.
• Programming experience with JavaScript/TypeScript, Python, C# or equivalent for automation, tooling and test data support.
• Experience with Agile Scrum development practices.
• Experience integrating automated tests into CI/CD pipelines (Azure DevOps, GitHub Actions).
• Experience testing distributed systems, backend services, APIs, integration-heavy workflows and security-sensitive systems.
• Experience in testing AI/LLM-powered applications, prompt-driven workflows, AI service APIs or agentic systems.

Responsibilities

• Own QA planning and delivery for AI-related features, services and workflows.
• Design, build and maintain automated test coverage for AI-enabled applications, APIs, backend services, data workflows and agent-driven user journeys.
• Test agentic systems, including tool selection, routing logic, orchestration, context injection, multi-step execution, failure handling and human-in-the-loop controls.
• Validate role-based access, permission boundaries, tenant isolation, identity propagation and secure tool execution across multi-tenant workflows.
• Design test strategies for context-aware behaviour where outcomes vary by tenant configuration, user permissions, feature flags, licensing and application state.
• Validate configurable AI behaviour, including prompt changes, tenant-specific configuration, tool configuration, versioned AI settings and rollback scenarios.
• Use risk-based testing, quality rubrics, LLM-as-a-judge, and repeatable evaluation scenarios to assess AI output quality, consistency, groundedness, hallucination risk and safe failure handling.
• Integrate automated checks and regression coverage into CI/CD pipelines and report quality risks, release readiness and defect trends clearly.
• Use observability and telemetry to investigate AI behaviour, agent traces, tool execution, failures, latency and production incidents.
• Work independently with strong ownership, proactive risk management and clear written communication across limited UK overlap.

Skills

QA fundamentals

Automation testing

Agentic/AI testing

Agile Scrum

Clear written communication

Tools

Playwright

Selenium

Cypress

Postman

REST Assured

Job description

ConnectOS is seeking an Automation Engineer to own QA for AI-enabled features within a Data and AI team. The role focuses on automated testing, AI service quality, agent workflow validation, and multi-tenant security across a distributed UK and Philippines pattern.

You will design and maintain automated test coverage for AI-enabled applications, APIs, and backend services, and work on agent-driven user journeys with strong ownership and proactive risk management.