AI Automation Tester (Hybrid Setup)

blaseek; Manila; On-site; PHP 3,616,636 - 4,822,182; Full time

Job summary

blaseek is searching for an experienced AI Automation Tester to ensure the quality and scalability of AI-powered solutions. You will design and maintain automated testing frameworks and collaborate with various teams to improve solution quality.

Ideal candidates will have over 5 years of experience in Software Quality Assurance with strong skills in automation testing tools like Selenium and Python. Responsibilities include creating test cases for Generative AI and contributing to AI testing standards.

Qualifications

• 5+ years of experience in Software Quality Assurance, Test Automation, or Quality Engineering.
• Strong hands-on experience with automation testing tools.
• Experience in developing automated test scripts using Python or similar.

Responsibilities

• Design, develop, and maintain automated test frameworks.
• Create and execute automated and manual test cases for AI applications.
• Collaborate with teams to improve solution quality.

Skills

Test Automation

API Testing

Python

Selenium

Automation Frameworks

Tools

Selenium

Azure DevOps

Cypress

Robot Framework

Job description

Overview

We are seeking an experienced AI Automation Tester to join our AI Center of Excellence (AI COE). This role will be responsible for ensuring the quality, reliability, and scalability of AI-powered solutions across the organization.

Key Responsibilities

• Design, develop, and maintain automated test frameworks for AI-enabled applications, APIs, and enterprise platforms.
• Create and execute automated and manual test cases for Generative AI, chatbot, copilot, and intelligent automation solutions.
• Validate AI-generated responses for accuracy, consistency, relevance, and compliance with business requirements.
• Develop test scenarios covering prompt validation, response quality, contextual understanding, and conversational flows.
• Identify and document hallucinations, inaccuracies, bias, safety concerns, and other AI-related quality issues.
• Validate Retrieval-Augmented Generation (RAG) solutions and ensure responses are aligned with approved knowledge sources.
• Perform functional, integration, regression, and end-to-end testing across AI-enabled business processes.
• Test AI workflows that integrate with enterprise systems, APIs, databases, and third-party applications.
• Contribute to AI testing standards, methodologies, governance, and best practices within the AI COE.
• Collaborate with AI Engineers, Data Scientists, Product Owners, Developers, and Business Stakeholders to improve solution quality.
• Support continuous integration and deployment processes by integrating automated tests into CI/CD pipelines.
• Produce testing documentation, quality metrics, and recommendations for AI solution improvements.

Required Qualifications

• 5+ years of experience in Software Quality Assurance, Test Automation, or Quality Engineering.
• Strong hands-on experience with automation testing tools such as Selenium, Playwright, Cypress, Robot Framework, or equivalent.
• Experience developing automated test scripts using Python, Java, JavaScript, or similar languages.
• Strong experience in API testing and validation of RESTful services.
• Solid SQL and database testing skills.
• Experience working with CI/CD tools such as Azure DevOps, Jenkins, GitHub Actions, or GitLab.
• Strong understanding of software testing methodologies, automation frameworks, and Agile delivery practices.
• Experience designing and executing end-to-end test strategies for complex applications.

Preferred Qualifications

• Experience testing AI-powered applications, chatbots, copilots, virtual assistants, or Generative AI solutions.
• Exposure to LLM platforms such as OpenAI, Microsoft Azure AI, Anthropic Claude, or Google Gemini.
• Understanding of prompt engineering, AI evaluation techniques, and LLM testing methodologies.
• Experience validating RAG solutions and AI knowledge-based systems.
• Familiarity with AI testing and evaluation tools such as Promptfoo, DeepEval, or LangSmith.
• Experience working with cloud platforms such as Azure, AWS, or Google Cloud.