Lead QA Engineer

Alivo; Portland, Northern (ME, KY); Hybrid; USD 70,000 - 110,000; Full time

Benefits offered by this job

Health insurance

Vision insurance

Dental insurance

Employee equity

401(k)

Job summary

Alivo is seeking a detail-oriented QA Engineer to own testing, release validation, and AI conversation quality across the platform. You will develop and run automated test suites, manually validate new features, and ensure releases meet high standards before customers see them.

You will evaluate the quality of AI-driven phone and chat conversations, identify issues, and help define quality standards and acceptance criteria for AI-powered interactions.

Qualifications

• 2–5+ years of QA, software testing, product operations, or similar experience.
• Strong attention to detail and a methodical approach to testing.
• Excellent written and verbal communication skills.
• Ability to identify edge cases, failure modes, and customer experience issues.
• Experience creating clear bug reports and testing documentation.
• Comfortable working in a fast-paced, evolving startup environment.

Responsibilities

• Own Product Testing: develop, run, and maintain automated test suites; manually test new features; perform exploratory testing; validate in staging environments; execute release checklists; verify bug fixes.
• Drive Product Quality & Release Readiness: review feature implementations and release candidates; partner with Engineering to assess risks; file clear bug reports; help establish manual testing processes; serve as a quality gate for releases.
• Own AI Conversation Quality: evaluate AI-powered conversations for naturalness, accuracy, usefulness; assess alignment with customer goals; review flow and outcomes; identify failures and edge cases; define acceptance criteria.
• Build AI Evaluation Processes: develop conversation evaluation datasets and testing scenarios; create scoring rubrics and quality frameworks; track quality and identify regressions; provide actionable recommendations to improve prompts and AI behavior.
• Collaborate Cross-Functionally: work with Engineering, Product, Customer Success, and Leadership; provide feedback to improve product quality and outcomes; prioritize improvements by customer impact.

Skills

Attention to detail

Analytical thinking

Communication skills

Bug reporting

Problem solving

Fast-paced environment

Tools

Jira

TestRail

Linear

Postman

Job description

About Alivo

At Alivo, we’re transforming how businesses connect with their customers. Using cutting-edge AI, we help companies deliver instant, intelligent, and personalized communication that drives growth and customer loyalty. Our mission is simple: empower businesses to thrive by making every interaction unforgettable.

Who We’re Looking For

We are looking for a highly detail-oriented QA Engineer to own testing, release validation, and AI conversation quality across the Alivo platform.

You will be in charge of developing and running our automated test suite as well as (where needed) manually validating new features, identifying edge cases, evaluating user experience, and ensuring releases meet a high standard before reaching customers.

A core part of this role is evaluating the quality of AI-driven phone and chat conversations. You will help determine whether our AI communicates naturally, accurately, and effectively while identifying opportunities to improve customer interactions and overall product quality.

This role is critical to our success. You will serve as the final quality gate before releases ship and help define what great AI-powered customer communication should look like.

If you thrive in fast-paced environments, enjoy uncovering problems before customers do, and have strong instincts for quality, this role is for you.

Responsibilities

• Own Product Testing
  • Develop, run, and maintain automated test suites
  • Manually test new features, workflows, and user experiences before release
  • Perform exploratory testing to identify bugs, usability issues, and edge cases
  • Validate functionality across staging and production-like environments
  • Execute structured test plans and release checklists to ensure consistent quality
  • Verify bug fixes and regression testing across product releases
• Drive Product Quality & Release Readiness
  • Review feature implementations and release candidates before deployment
  • Partner with Engineering to identify risks and improve release confidence
  • File clear, reproducible bug reports with detailed documentation
  • Help establish and improve manual testing processes as the company grows
  • Serve as a trusted quality gate for production releases
• Own AI Conversation Quality
  • Evaluate AI-powered phone and chat conversations for quality and effectiveness
  • Assess whether conversations are natural, accurate, helpful, and aligned with customer goals
  • Review objection handling, conversational flow, and customer experience outcomes
  • Identify failure modes, edge cases, hallucinations, and off-script behavior
  • Help define quality standards and acceptance criteria for AI-powered interactions
• Build AI Evaluation Processes
  • Develop and maintain conversation evaluation datasets and testing scenarios
  • Create scoring rubrics and quality frameworks for measuring AI performanceTrack AI quality over time and identify regressions or performance changes
  • Provide actionable recommendations that improve prompts, workflows, and AI behavior
• Collaborate Cross-Functionally
  • Work closely with Engineering, Product, Customer Success, and Leadership
  • Provide feedback that improves both product quality and customer outcomes
  • Help prioritize quality improvements based on customer impact and business goals
  • Support a culture of quality throughout the organization

Qualifications

• What You Bring
  • 2–5+ years of QA, software testing, product operations, or similar experience
  • Strong attention to detail and a methodical approach to testing
  • Excellent written and verbal communication skills
  • Ability to identify edge cases, failure modes, and customer experience issues
  • Strong problem-solving and analytical thinking skills
  • Experience creating clear bug reports and testing documentation
  • Comfortable working in a fast-paced, evolving startup environment
• Who You Are
  • Highly detail-oriented and naturally curious
  • A strong critical thinker who enjoys finding problems before customers do
  • Comfortable making judgment calls about quality and user experience
  • Organized, reliable, and highly accountable
  • Able to communicate findings clearly and objectively
  • Passionate about creating exceptional customer experiences
• Bonus Points For
  • Experience evaluating AI, LLM, or conversational AI systems
  • Experience with test management or bug tracking tools such as Linear, Jira, or TestRail
  • Experience with API testing tools such as Postman
  • Basic SQL, scripting, or technical troubleshooting skills
  • Experience in customer-facing, sales, or support roles
  • Exposure to AI-powered products or machine learning systems
  • Background in SaaS products or high-growth startup environments

• Company Sponsored Insurance Offerings
  • Health
  • Vision
  • Dental
• Long Term Financial Benefits
  • Employee equity
  • 401(k)
• Time to Recharge
  • Unlimited paid time off
  • Paid sick time