QA Engineer

ArrowCore Group; India; On-site; INR 800,000 - 1,100,000; Full time

Job summary

ArrowCore Group is seeking a QA Engineer II to ensure the quality of Conversational AI solutions. The role focuses on semantic/behavioral evaluation for non-deterministic agents, alongside traditional test design and execution.

The candidate should design benchmarks for LLMs, perform manual and automated testing, and contribute to test automation while collaborating with product teams. Experience with Bash, Python, Playwright/Selenium, and JMeter is preferred.

Qualifications

• 3+ years of experience in software/product quality assurance.
• Confident usage of Bash and Python.
• Experience with manual testing and test case execution.
• Basic familiarity with automated testing frameworks such as Playwright or Selenium.
• Experience with testing web protocols (WebSocket message payload).
• Basic knowledge of load and performance testing (e.g., JMeter).
• Strong analytical and problem-solving abilities.
• Systems thinking.
• Familiarity with prompt evaluation, function-calling, and tool-use testing.

Responsibilities

• Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behavior and responses.
• Functional & Regression Testing: Grow benchmarks and create new ones as new functionality or issues are uncovered.
• Benchmarking Automation Support: Contribute to streamlining the benchmarking process in alignment with product development work.
• Collaboration: Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.
• Defect Identification & Reporting: Log detailed defect reports and work with engineering teams to ensure timely resolution.
• Process & Traceability: Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
• Release Reporting: Produce per-release quality reports as benchmarks evolve and grow over time.
• Performance & Load Testing Support: Assist in load and performance testing using tools such as JMeter.
• Agile Testing & Continuous Improvement: Participate in Agile development cycles, contributing to continuous process improvements.

Skills

Bash

Python

Manual testing

Playwright

Selenium

WebSocket testing

Load testing

Analytical thinking

Systems thinking

Prompt evaluation

Tools

Playwright

Selenium

JMeter

Job description

The QA Engineer II plays a crucial role in ensuring the quality and reliability of our Conversational AI solutions. Unlike deterministic software, the AI agents under test are non-deterministic, so benchmarking emphasizes semantic/behavioral evaluation in addition to exact-match assertions.

The ideal candidate has a strong foundation in software testing methodologies, experience with test case design and execution, and familiarity with automation frameworks. This role is ideal for detail-oriented professionals who are eager to learn, contribute to test automation, and help drive continuous improvements in software quality.

Responsibilities

• Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behavior and responses.
• Functional & Regression Testing: Grow benchmarks and create new ones as new functionality or issues are uncovered.
• Benchmarking Automation Support: Contribute to streamlining the benchmarking process in alignment with product development work.
• Collaboration: Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.
• Defect Identification & Reporting: Log detailed defect reports and work with engineering teams to ensure timely resolution.
• Process & Traceability: Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
• Release Reporting: Produce per-release quality reports as benchmarks evolve and grow over time.
• Performance & Load Testing Support: Assist in load and performance testing using tools such as JMeter.
• Agile Testing & Continuous Improvement: Participate in Agile development cycles, contributing to continuous process improvements.

Qualifications

Basic Qualifications (Non-Negotiable)

• 3+ years of experience in software/product quality assurance.
• Confident usage of Bash and Python.
• Experience with manual testing and test case execution.
• Basic familiarity with automated testing frameworks such as Playwright or Selenium.
• Experience with testing web protocols (WebSocket message payload)
• Basic knowledge of load and performance testing (e.g., JMeter).
• Strong analytical and problem-solving abilities.
• Systems thinking.
• Familiarity with prompt evaluation, function-calling, and tool-use testing.

Preferred Skills

• Exposure to test automation scripting using Python.
• Experience with CI/CD integration.
• Familiarity with Conversational AI platforms and NLP-based applications.
• Experience with test management and result tracking.
• Experience testing conversational-agent systems and evaluating non-deterministic outputs.

General Requirements

• Eager to learn and grow in the field of QA and test automation.
• Strong collaborative mindset and ability to work effectively in a global team environment.
• Ability to adapt to fast-paced development cycles and evolving requirements.