A complete application in a minute
Tailored resume and cover letter, ready to send.
BruntWork is seeking a detail-oriented Senior QA Engineer (AI & Automation Testing) to join its technical team. You will evaluate LLMs, conversational flows, and system integrations, designing and executing tests to benchmark AI output quality across web and mobile platforms.
You will build automated UI and API tests, perform adversarial and safety testing, and integrate AI quality gates into CI/CD pipelines.
We are seeking a detail-oriented Senior QA Engineer (AI & Automation Testing)to join our technical team in ensuring the reliability, accuracy, and security of Candid Chat AI across both web and mobile platforms.
In this role, you will go beyond traditional software testing by actively evaluating Large Language Models (LLMs), conversational flows, and system integrations. You will design, execute, and automate tests to benchmark AI output quality, conduct rigorous adversarial and safety testing, and integrate automated quality gates into our CI/CD pipelines.
Monday to Friday, 10:00 AM to 7:00 PM (Australian Eastern Time, 40 hours per week)
AI & Conversational Testing: Test end-to-end conversational AI flows across web and mobile interfaces to ensure seamless user experiences.
Model Evaluation & Quality Assurance: Systematically evaluate AI responses for factual accuracy, context relevance, tone consistency, and hallucination detection.
Adversarial & Safety Testing: Perform red-teaming and adversarial testing, including prompt injection, jailbreaking attempts, and guardrail verification to ensure model safety and compliance.
Regression Testing: Design and execute regression test suites whenever model versions, system prompts, or backend configurations are updated.
Test Automation: Build and maintain UI and API automated testing frameworks for frontend applications and underlying backend services.
Benchmarking & Datasets: Construct and maintain benchmark test datasets to track, measure, and score AI response quality over time.
CI/CD Integration: Integrate automated AI quality and regression checks directly into the continuous integration and deployment (CI/CD) pipelines.
Bug Documentation & Escalation: Identify, isolate, and document AI-specific bugs, model drift, and system anomalies with clear reproduction steps for Machine Learning and Engineering teams.
Experience: 4+ years of experience in Software Quality Assurance, with strong manual and automation testing experience.
UI & API Testing: Proven experience in UI testing, API testing, and test automation using tools such as Selenium, Playwright, Cypress, Postman, REST Assured, or Python.
Technical Proficiency: Strong scripting skills in Python or JavaScript, with experience in CI/CD tools such as GitHub Actions, GitLab CI, or Jenkins.
Adaptability to AI Testing: Ability to quickly learn and adapt testing approaches to conversational AI, chatbots, and LLM-powered applications, including evaluating response accuracy, relevance, consistency, and hallucinations.
Communication & Detail: Strong analytical and problem-solving skills with the ability to communicate complex model behavior issues clearly to engineering teams.
Industry background: Background in financial technology, SaaS, or high-growth tech environments.
Conversational AI Experience: Previous experience testing conversational AI, chatbots, NLP models, or LLM-powered applications.
LLM Evaluation & Safety Testing: Hands-on experience evaluating hallucinations, prompt safety, model responses, and guardrails.
Adversarial Testing: Familiarity with common LLM vulnerabilities, prompt injection techniques, jailbreaking attempts, and AI security testing.
Specialized Testing Experience: Experience testing AI-driven conversational flows across web and mobile applications, including end-to-end testing of complex interactions.
Permanent work from home
Immediate hiring
Health Insurance Coverage for eligible locations