QA Automation Engineer

Chatoly Assistant; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Chatoly Assistant is seeking a QA automation engineer to own end-to-end web testing across multiple surfaces, including AI response evaluation considerations. You will build and maintain tests for the dashboard, widget, and public site, and design regression strategies with meaningful correctness criteria beyond fixed strings.

You will also validate integrations in partner sandboxes, perform cross-browser and accessibility checks, and seek speed and stability improvements to keep the suite

Qualifications

• At least three years of QA automation experience for web applications.
• Practical experience with Playwright, Cypress, or a comparable browser automation framework.
• Ability to read and write application code, not only test-specific scripts.
• Methodical debugging and defect-isolation habits.
• Sound judgment about which scenarios benefit from automation and which do not.

Responsibilities

• Develop and maintain end-to-end tests for the dashboard, embedded widget, and public-facing site.
• Design regression strategies for AI-generated responses using meaningful correctness criteria.
• Exercise integrations in partner sandboxes and identify compatibility problems before customers
• Include desktop, mobile, cross-browser, and accessibility checks in release process.
• Improve test execution speed, stability, and diagnostic value for team trust.
• Reproduce, isolate, and document defects clearly to support efficient fixes.

Skills

QA automation

Web application testing

Code reading/writing

Debugging

Automation strategy

Tools

Playwright

Cypress

Job description

Role overview

Build the automated quality foundation that enables a small product team to release quickly without disrupting customer conversations. The role covers end-to-end web testing across multiple surfaces, partner integration validation, cross-browser and accessibility coverage, and the more nuanced challenge of evaluating AI responses whose correctness may be contextual rather than an exact text match.

Responsibilities

• Develop and maintain end-to-end tests for the dashboard, embedded widget, and public-facing site.
• Design regression strategies for AI-generated responses using meaningful correctness criteria rather than fixed-string comparisons alone.
• Exercise integrations in partner sandboxes and identify compatibility problems before they reach customers.
• Include desktop, mobile, cross-browser, and accessibility checks in the release process.
• Improve test execution speed, stability, and diagnostic value so the suite remains trusted by the team.
• Reproduce, isolate, and document defects clearly enough to support efficient fixes.

Requirements

• At least three years of QA automation experience for web applications.
• Practical experience with Playwright, Cypress, or a comparable browser automation framework.
• Ability to read and write application code, not only test-specific scripts.
• Methodical debugging and defect-isolation habits.
• Sound judgment about which scenarios benefit from automation and which do not.

Nice to have

• Experience evaluating LLMs or other non-deterministic systems.
• Accessibility auditing experience against WCAG.
• Background in performance or load testing.