NLP QA Engineer

BruntWork; Philippines; A distancia; PHP 600.000 - 1.200.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Permanent work from home

Immediate hiring

Health Insurance Coverage for eligible

Descripción de la vacante

BruntWork is seeking a detailed NLP QA Engineer to ensure reliability, accuracy, and safety of Candid Chat AI across web and mobile platforms. You will design, execute, and automate tests for LLMs, conversational flows, and system integrations.

You will work within CI/CD pipelines, perform adversarial and safety testing, and document AI-specific issues with clear reproduction steps for ML and engineering teams. 3+ years in QA for conversational AI and strong API/UI automation skills are required.

Formación

• 3+ years of Software Quality Assurance experience, focusing on Conversational AI or LLM-powered apps.
• Proven track record in testing NLP models, prompt safety, and hallucination evaluation.

Responsabilidades

• Test end-to-end conversational AI flows across web and mobile interfaces.
• Evaluate AI responses for factual accuracy, context relevance, and tone consistency.
• Perform red-teaming and adversarial testing, including prompt injection and guardrail verification.
• Design and execute regression test suites when models or backends are updated.
• Build and maintain UI and API automated testing frameworks.
• Create benchmark datasets to track AI output quality over time.
• Integrate automated AI quality and regression checks into CI/CD pipelines.
• Document AI-specific bugs with reproduction steps for ML and Engineering teams.

Conocimientos

NLP QA

API testing

UI automation

Adversarial testing

Documentation & analysis

Communication

Herramientas

Postman

REST Assured

Python

Selenium

Playwright

Appium

Cypress

Descripción del empleo

We are seeking a detail-oriented NLP QA Engineer to join our technical team in ensuring the reliability, accuracy, and security of Candid Chat AI across both web and mobile platforms.

In this role, you will go beyond traditional software testing by actively evaluating Large Language Models (LLMs), conversational flows, and system integrations. You will design, execute, and automate tests to benchmark AI output quality, conduct rigorous adversarial and safety testing, and integrate automated quality gates into our CI/CD pipelines.

Schedule:

• Monday to Friday, flexible schedule within the client’s business hours (Australian Eastern Time, 40 hours per week)

Key Responsibilities:

• AI & Conversational Testing: Test end-to-end conversational AI flows across web and mobile interfaces to ensure seamless user experiences.
• Model Evaluation & Quality Assurance: Systematically evaluate AI responses for factual accuracy, context relevance, tone consistency, and hallucination detection.
• Adversarial & Safety Testing: Perform red-teaming and adversarial testing, including prompt injection, jailbreaking attempts, and guardrail verification to ensure model safety and compliance.
• Regression Testing: Design and execute regression test suites whenever model versions, system prompts, or backend configurations are updated.
• Test Automation: Build and maintain UI and API automated testing frameworks for frontend applications and underlying backend services.
• Benchmarking & Datasets: Construct and maintain benchmark test datasets to track, measure, and score AI response quality over time.
• CI/CD Integration: Integrate automated AI quality and regression checks directly into the continuous integration and deployment (CI/CD) pipelines.
• Bug Documentation & Escalation: Identify, isolate, and document AI-specific bugs, model drift, and system anomalies with clear reproduction steps for Machine Learning and Engineering teams.

Key Requirements:

• Experience: 3+ years of experience in Software Quality Assurance, with a focus on Conversational AI, Chatbots, or LLM-powered applications.
• Testing Focus: Proven track record in testing natural language processing (NLP) models, prompt safety, and hallucination evaluation.
• Automation Skills: Hands-on experience with API testing (Postman, REST Assured, or Python) and UI automation tools (Selenium, Playwright, Appium, or Cypress).
• Adversarial Knowledge: Familiarity with common LLM vulnerabilities, prompt injection techniques, and security testing principles.
• Communication & Detail: Strong analytical and problem-solving skills with the ability to communicate complex model behavior issues clearly to engineering teams.
• Permanent work from home
• Immediate hiring
• Health Insurance Coverage for eligible locations