Automation Test Engineer with AI

Nxtwave Disruptive Technologies(Hiring for a Client); United States; Remote; USD 85,000 - 120,000; Full time

Job summary

Nxtwave Disruptive Technologies(Hiring for a Client) is seeking a QA Engineer II to ensure the quality, reliability, and performance of Conversational AI applications. The role requires strong software testing fundamentals, hands-on Python and Bash scripting, and familiarity with Playwright or Selenium for automation.

The position involves testing AI agents, evaluating their behavior, and maintaining reliable benchmarks for continuous quality improvements within an Agile environment.

Qualifications

• Strong fundamentals of software testing and QA processes.
• Experience with Python and Bash scripting for test automation.
• Familiarity with Playwright or Selenium for automation.

Responsibilities

• Design, create, manage, and maintain test benchmarks for AI agents and LLM-based applications.
• Perform functional, regression, and manual testing to validate behavior and identify defects.
• Evaluate AI-generated responses based on semantic accuracy and task completion.
• Support test automation and benchmarking using Python scripts.
• Test WebSocket communications and validate message payloads.
• Document defects and cooperate with developers and QA teams.
• Maintain test cases, plans, and coverage documentation.
• Prepare quality reports for releases and track improvements in test coverage.
• Support performance and load testing using JMeter.
• Participate in Agile development cycles and drive quality improvements.

Skills

Python

Bash scripting

Manual testing

Test case design

Debugging

Tools

Playwright

Selenium

JMeter

Job description

We are looking for a QA Engineer II to ensure the quality, reliability, and performance of Conversational AI applications. The ideal candidate should have strong software testing fundamentals, hands-on experience with Python and Bash scripting, and familiarity with test automation frameworks such as Playwright or Selenium.

The role involves testing AI agents whose responses may vary, evaluating their behavior, validating functionality, and maintaining reliable benchmarks for continuous quality improvement.

Key Responsibilities

• Design, create, manage, and maintain test benchmarks for AI agents and Large Language Model (LLM)-based applications.
• Perform functional, regression, and manual testing to validate application behavior and identify defects.
• Evaluate AI-generated responses based on semantic accuracy, expected behavior, and task completion rather than exact-match assertions alone.
• Support test automation and benchmarking using Python scripts.
• Test WebSocket communications and validate message payloads.
• Identify, document, track, and report defects while collaborating with developers and QA teams.
• Maintain test cases, test plans, requirement traceability, and test coverage documentation.
• Prepare quality reports for releases and track improvements in test coverage.
• Support performance and load testing using tools such as JMeter.
• Participate in Agile development cycles and contribute to continuous quality improvements.

Mandatory Skills and Qualifications

• Minimum 3+ years of experience in software or product quality assurance.
• Strong working knowledge of Python and Bash scripting.
• Hands-on experience in manual testing, test case design, and test execution.
• Basic familiarity with automation frameworks such as Playwright or Selenium.
• Experience testing WebSocket communications and message payloads.
• Basic understanding of performance and load testing tools such as JMeter.
• Familiarity with AI testing concepts, including prompt evaluation, function calling, and tool-use testing.
• Strong analytical, problem-solving, and debugging skills.
• Ability to document test results and collaborate effectively with cross-functional teams.

Preferred Skills

• Experience writing Python-based test automation scripts.
• Exposure to CI/CD pipelines and automated test integration.
• Familiarity with Conversational AI, NLP-based applications, and LLM-powered systems.
• Experience with test management tools, benchmark tracking, and test result reporting.
• Exposure to evaluating non-deterministic AI outputs and conversational agent behavior.