QA Automation Engineer

Chatoly Assistant; United States; Remote; USD 90,000 - 130,000; Full time

Job summary

Chatoly Assistant is seeking a QA automation engineer to build the automated quality foundation that lets a small product team release quickly without disrupting customer conversations. You will own end-to-end web tests across the dashboard, embedded widget, and public site, design regression strategies for AI-generated responses, and ensure cross-browser and accessibility coverage.

The role emphasizes fast diagnosis, reproducibility, and clear defect isolation to keep the suite trusted by the

Qualifications

• Three years or more of QA automation experience for web applications.
• Experience reading and writing application code, not just test scripts.
• Ability to design robust regression strategies and debugging habits.

Responsibilities

• Develop and maintain end-to-end tests for the dashboard, embedded widget, and public-facing site.
• Design regression strategies for AI-generated responses using meaningful correctness criteria.
• Exercise integrations in partner sandboxes and identify compatibility problems before customers see them.
• Include desktop, mobile, cross-browser, and accessibility checks in the release process.
• Improve test execution speed, stability, and diagnostic value to keep the suite trusted by the team.
• Reproduce, isolate, and document defects clearly to support efficient fixes.

Skills

QA automation

Web testing

Playwright

Cypress

Browser automation

Test strategy

Tools

Playwright

Cypress

Job description

Role overview

Build the automated quality foundation that enables a small product team to release quickly without disrupting customer conversations. The role covers end-to-end web testing across multiple surfaces, partner integration validation, cross-browser and accessibility coverage, and the more nuanced challenge of evaluating AI responses whose correctness may be contextual rather than an exact text match.

Responsibilities

• Develop and maintain end-to-end tests for the dashboard, embedded widget, and public-facing site.
• Design regression strategies for AI-generated responses using meaningful correctness criteria rather than fixed-string comparisons alone.
• Exercise integrations in partner sandboxes and identify compatibility problems before they reach customers.
• Include desktop, mobile, cross-browser, and accessibility checks in the release process.
• Improve test execution speed, stability, and diagnostic value so the suite remains trusted by the team.
• Reproduce, isolate, and document defects clearly enough to support efficient fixes.

Requirements

• At least three years of QA automation experience for web applications.
• Practical experience with Playwright, Cypress, or a comparable browser automation framework.
• Ability to read and write application code, not only test-specific scripts.
• Methodical debugging and defect-isolation habits.
• Sound judgment about which scenarios benefit from automation and which do not.

Nice to have

• Experience evaluating LLMs or other non-deterministic systems.
• Accessibility auditing experience against WCAG.
• Background in performance or load testing.