WFH Senior QA Engineer (Conversational AI)

BruntWork

Metro Manila

Remote

PHP 750,000 - 1,100,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Permanent work from home
Immediate hiring
Health Insurance Coverage for eligible

Job summary

BruntWork is seeking a detail-oriented Senior QA Engineer (AI & Automation Testing) to join its technical team. You will evaluate LLMs, conversational flows, and system integrations, designing and executing tests to benchmark AI output quality across web and mobile platforms.

You will build automated UI and API tests, perform adversarial and safety testing, and integrate AI quality gates into CI/CD pipelines.

Qualifications

  • 4+ years of Software QA experience with both manual and automated testing.
  • Proven UI and API testing experience with automation tooling.
  • Strong scripting skills in Python or JavaScript.
  • Experience with CI/CD tools (GitHub Actions, GitLab CI, Jenkins).

Responsibilities

  • Test end-to-end conversational AI flows across web and mobile.
  • Evaluate AI responses for accuracy, relevance, tone, and hallucinations.
  • Perform adversarial and safety testing including prompt injections.
  • Design and execute regression tests for model/version updates.
  • Build and maintain automated UI and API testing frameworks.
  • Create benchmark datasets to track AI output quality.
  • Integrate AI quality checks into CI/CD pipelines.
  • Document AI-specific bugs and model drift with clear reproduction steps.

Skills

UI testing
API testing
Automation testing
Scripting in Python/JS
CI/CD familiarity
Communication & detail

Tools

Selenium
Playwright
Cypress
Postman
REST Assured
Python
JavaScript
GitHub Actions
GitLab CI
Jenkins

Job description

About the Role:

We are seeking a detail-oriented Senior QA Engineer (AI & Automation Testing)to join our technical team in ensuring the reliability, accuracy, and security of Candid Chat AI across both web and mobile platforms.

In this role, you will go beyond traditional software testing by actively evaluating Large Language Models (LLMs), conversational flows, and system integrations. You will design, execute, and automate tests to benchmark AI output quality, conduct rigorous adversarial and safety testing, and integrate automated quality gates into our CI/CD pipelines.

Schedule:
  • Monday to Friday, 10:00 AM to 7:00 PM (Australian Eastern Time, 40 hours per week)

Key Responsibilities:
  • AI & Conversational Testing: Test end-to-end conversational AI flows across web and mobile interfaces to ensure seamless user experiences.

  • Model Evaluation & Quality Assurance: Systematically evaluate AI responses for factual accuracy, context relevance, tone consistency, and hallucination detection.

  • Adversarial & Safety Testing: Perform red-teaming and adversarial testing, including prompt injection, jailbreaking attempts, and guardrail verification to ensure model safety and compliance.

  • Regression Testing: Design and execute regression test suites whenever model versions, system prompts, or backend configurations are updated.

  • Test Automation: Build and maintain UI and API automated testing frameworks for frontend applications and underlying backend services.

  • Benchmarking & Datasets: Construct and maintain benchmark test datasets to track, measure, and score AI response quality over time.

  • CI/CD Integration: Integrate automated AI quality and regression checks directly into the continuous integration and deployment (CI/CD) pipelines.

  • Bug Documentation & Escalation: Identify, isolate, and document AI-specific bugs, model drift, and system anomalies with clear reproduction steps for Machine Learning and Engineering teams.

Must-have Requirements:
  • Experience: 4+ years of experience in Software Quality Assurance, with strong manual and automation testing experience.

  • UI & API Testing: Proven experience in UI testing, API testing, and test automation using tools such as Selenium, Playwright, Cypress, Postman, REST Assured, or Python.

  • Technical Proficiency: Strong scripting skills in Python or JavaScript, with experience in CI/CD tools such as GitHub Actions, GitLab CI, or Jenkins.

  • Adaptability to AI Testing: Ability to quickly learn and adapt testing approaches to conversational AI, chatbots, and LLM-powered applications, including evaluating response accuracy, relevance, consistency, and hallucinations.

  • Communication & Detail: Strong analytical and problem-solving skills with the ability to communicate complex model behavior issues clearly to engineering teams.

Nice to have:
  • Industry background: Background in financial technology, SaaS, or high-growth tech environments.

  • Conversational AI Experience: Previous experience testing conversational AI, chatbots, NLP models, or LLM-powered applications.

  • LLM Evaluation & Safety Testing: Hands-on experience evaluating hallucinations, prompt safety, model responses, and guardrails.

  • Adversarial Testing: Familiarity with common LLM vulnerabilities, prompt injection techniques, jailbreaking attempts, and AI security testing.

  • Specialized Testing Experience: Experience testing AI-driven conversational flows across web and mobile applications, including end-to-end testing of complex interactions.

Independent Contractor Perks
  • Permanent work from home

  • Immediate hiring

  • Health Insurance Coverage for eligible locations

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead QA Engineer (AI Voice)
Lead QA Engineer (AI Voice)

Lingaro Group Philippines • Manila

On-site
PHP 600,000 - 900,000
QA / AI Evaluation & Research Engineer
QA / AI Evaluation & Research Engineer

Datamatics Careers- Philippines • Pasig

On-site
PHP 1,200,000 - 2,100,000
AI Automation Tester (Hybrid Setup)
AI Automation Tester (Hybrid Setup)

blaseek • Manila

On-site
PHP 3,616,636 - 4,822,182
Quality Assurance/QA Specialist
Quality Assurance/QA Specialist

opencall.ai (yc w24) • Philippines

On-site
PHP 622,000 - 745,000
Senior Automation QA (C#) Engineer
Senior Automation QA (C#) Engineer

N-iX LTD • España

Hybrid
PHP 1,513,317 - 2,723,971
Flexible working format
Competitive salary
Personalized career growth
+3
Software Test Engineer
Software Test Engineer

Probe CX • Metro Manila

Remote
PHP 480,000 - 600,000
AI Solution Engineer
AI Solution Engineer

Hyre • Quezon City

Hybrid
PHP 3,472,000 - 6,944,000
Paid time off
Hyre Academy access
Rate increases based on contract terms
Senior Quality Assurance Engineer
Senior Quality Assurance Engineer

Bynd • Manila

Hybrid
PHP 1,080,000 - 1,800,000
Competitive base salary
De Minimis Allowance
13th Month Pay
+4
Mid-level SDET (WFH)
Mid-level SDET (WFH)

TASQ • Philippines

Remote
PHP 360,000 - 540,000
AI Engineer (Generative AI)-Contractor
AI Engineer (Generative AI)-Contractor

Cloud Mile Corp. • Manila

On-site
PHP 892,800 - 1,339,200
Top locations