Quality Assurance Automation Engineer

Structurely; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote work

Job summary

21X is building the next generation of conversational AI infrastructure. We seek a highly technical Senior QA Automation Engineer to design, implement, and own our QA ecosystem, including AI-driven testing and release quality across a fast-moving enterprise platform.

You will lead automation across voice, SMS, RCS, and AI agents, build scalable testing frameworks, and mentor manual QA engineers as the team grows. This remote, full-time role demands independence and impact.

Qualifications

• 5+ years in software QA engineering with extensive test automation experience.
• Proven experience building QA automation frameworks from scratch.
• Strong programming skills in TypeScript, JavaScript, or Python.
• Hands-on experience with Playwright, Cypress, Selenium, or similar frameworks.
• Strong API testing experience, including REST and GraphQL.
• Experience implementing CI/CD testing pipelines.
• Experience testing distributed systems and complex integrations.
• Experience leading or mentoring QA engineers.
• Strong understanding of software architecture, debugging, and development workflows.
• Ability to operate independently in a fast-moving startup environment.

Responsibilities

• Architect and implement an automated testing framework from scratch.
• Develop end-to-end, integration, API, and regression testing.
• Integrate automated testing into CI/CD pipelines.
• Establish automated testing across development, staging, and production environments.
• Create reusable test frameworks that scale with our engineering organization.
• Implement intelligent test execution, reporting, and failure analysis.
• Develop automation to validate Voice AI, SMS & RCS, AI Agents, CRM Integrations, and Enterprise Workflows.
• Build automated regression suites for AI agent behavior and model updates.
• Leverage LLMs and AI-assisted testing to generate and maintain test cases.
• Create systems that detect regressions in AI performance.
• Lead and mentor manual QA engineers; establish QA standards and release validation processes.
• Own release quality with automated smoke tests and dashboards for coverage and trends.

Skills

TS/JS/Python programming

Tools

Playwright

Cypress

Selenium

Job description

21X is building the next generation of conversational AI infrastructure.

Our platform powers intelligent AI agents that communicate with customers across voice, SMS, RCS, email, and web. We help enterprises automate complex customer interactions, execute transactions, and deliver exceptional customer experiences at scale.

We’re growing across mortgage, aviation, sports, and other industries, with a strong focus on enterprise reliability, performance, and scalability.

Senior QA Automation Engineer to build and own our quality assurance function.

QA Automation Engineer / QA Lead

21X | Remote | Full-Time

About 21X

21X is building the next generation of conversational AI infrastructure.

Our platform powers intelligent AI agents that communicate with customers across voice, SMS, RCS, email, and web. We help enterprises automate complex customer interactions, execute transactions, and deliver exceptional customer experiences at scale.

We’re growing across mortgage, aviation, sports, and other industries, with a strong focus on enterprise reliability, performance, and scalability.

Senior QA Automation Engineer to build and own our quality assurance function.

The Opportunity

This isn’t a traditional QA position.

We’re looking for a highly technical, automation-first engineer who can design, implement, and lead our entire testing ecosystem.

You’ll work directly with engineering and product leadership to create a scalable QA infrastructure that enables rapid development without compromising quality.

Our ambition is to automate nearly everything that can reasonably be automated.

You’ll establish the frameworks, infrastructure, processes, and standards that make that possible.

As the company grows, you’ll also lead and manage manual QA engineers responsible for exploratory testing, complex scenarios, and validating AI behavior.

Key Responsibilities

1. Build Our QA Automation Infrastructure

• Architect and implement an automated testing framework from scratch.
• Develop comprehensive end-to-end, integration, API, and regression testing.
• Integrate automated testing into our CI/CD pipelines.
• Establish automated testing across development, staging, and production environments.
• Create reusable test frameworks that scale with our engineering organization.
• Implement intelligent test execution, reporting, and failure analysis.

2. Automate Testing Across Our AI Platform

Our technology presents unique testing challenges. You’ll develop automation to validate:

• Voice AI: Call flows, agent responses, interruptions, transfers, latency, and conversation outcomes.
• SMS & RCS: Message delivery, conversational workflows, rich messaging, and carrier integrations.
• AI Agents: Tool execution, prompt adherence, knowledge retrieval, conversational accuracy, and agent behavior.
• CRM Integrations: Salesforce and other enterprise integrations, data synchronization, and API reliability.
• Enterprise Workflows: Multi-step automation, appointment scheduling, transaction execution, and complex business logic.

You’ll also build automated regression suites for AI agent behavior to identify changes in conversational performance as models, prompts, and underlying infrastructure evolve.

3. Implement AI-Powered QA

We want someone who thinks beyond traditional test automation.

• Leverage LLMs and AI-assisted testing to generate and maintain test cases.
• Build automated conversation simulations across multiple channels.
• Develop evaluation frameworks for conversational AI quality.
• Implement automated scoring for agent accuracy, compliance, and task completion.
• Create systems that detect regressions in AI performance.
• Explore autonomous testing agents that can identify defects and validate new functionality.

4. Lead Manual QA Engineers

• Establish QA standards, test plans, and release validation processes.
• Lead and mentor manual QA engineers as the team grows.
• Delegate exploratory testing, edge-case validation, and complex user scenarios.
• Create clear defect reporting, severity classifications, and escalation procedures.
• Ensure manual testing complements automation rather than duplicating it.

5. Own Release Quality

• Establish measurable quality gates for production releases.
• Implement automated smoke tests and regression testing.
• Build dashboards for test coverage, defect trends, and release readiness.
• Partner with engineering leadership to reduce production defects.
• Continuously improve test reliability, execution speed, and automation coverage.

What We’re Looking For

Required Experience

• 5+ years in software QA engineering, with extensive test automation experience.
• Proven experience building QA automation frameworks from scratch.
• Strong programming skills in TypeScript, JavaScript, or Python.
• Hands-on experience with Playwright, Cypress, Selenium, or similar frameworks.
• Strong API testing experience, including REST and GraphQL.
• Experience implementing CI/CD testing pipelines.
• Experience testing distributed systems and complex integrations.
• Experience leading or mentoring QA engineers.
• Strong understanding of software architecture, debugging, and development workflows.
• Ability to operate independently in a fast-moving startup environment.

Highly Preferred

• Experience testing conversational AI, LLM applications, or AI agents.
• Familiarity with LLM evaluation frameworks such as LangSmith, Braintrust, or custom evaluation pipelines.
• Experience with voice technologies, telephony systems, SIP, or messaging APIs.
• Familiarity with AWS and cloud-native environments.
• Experience implementing load, performance, and reliability testing.
• Experience with automated testing of nondeterministic systems.

Our Ideal Candidate

You’re an engineer first and a QA professional second.

You write production-quality code, understand complex systems, and believe that repetitive manual testing should be automated wherever practical.

You don’t wait for someone to tell you what to test. You understand the architecture, identify risk, and build systems that catch problems before customers experience them.

You’re equally comfortable writing automation code, designing testing infrastructure, and leading engineers.

You thrive in environments where you can build something from scratch and take complete ownership of the outcome.

Success Metrics

I’d measure this hire against five outcomes:

• Defect prevention: Meaningful reduction in customer-reported defects.
• Automation coverage: 90%+ of critical, repeatable regression scenarios automated over time.
• Release confidence: Every production deployment passes defined automated quality gates.
• Testing efficiency: Significant reduction in the time required to validate a release.
• AI reliability: Automated detection of regressions in conversation outcomes, tool execution, latency, and agent behavior.