Sr QA Engineer, AI Agents & Agentic Workflows

QAD; Pune District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

QAD, Pune, seeks a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi-agent AI product with evolving agents. You will test agents, evaluate reasoning, tool use, orchestration and handoffs, and design evaluation frameworks and regression suites that catch non-deterministic AI failures.

Responsibilities include crafting end-to-end agentic scenarios, auditing outputs for correctness, and identifying cross-agent issues.

Qualifications

• Bachelor's degree in software engineering, computer science or a related field.
• 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or agentic workflows.
• Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and acceptance thresholds for AI responses.
• Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets, scoring rubrics).
• Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning, memory/context, RAG.
• Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.

Responsibilities

• Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows, including agent routing, tool/function calls, and multi-turn conversations.
• Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and completeness, using both human review and automated methods.
• Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated regression suites that run as agents, prompts, models and tools change.
• Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs, tool-call failures, context loss, looping or runaway behavior.
• Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
• Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and quantify them with metrics.
• Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool and orchestration causes.
• Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
• Document test strategy, eval results and quality trends clearly for global stakeholders.

Skills

QA testing

LLM evaluation

Regression suites

Automation tools

Education

Bachelor's in Software Engineering/CS

Tools

Selenium

Playwright

TestComplete

Job description

QAD is seeking a Sr QA Engineer to own the continuous validation and quality of Champion Assist, a multi-agent AI product with ~20 evolving agents. You will test the agents themselves: their reasoning, tool use, orchestration and handoffs, and the quality of their responses. This goes beyond UI and API checks. You will design evaluation frameworks and regression suites that catch failures unique to non-deterministic, LLM-driven systems.

Responsibilities

• Design, run and maintain end-to-end business scenarios that exercise multi-step agentic workflows, including agent routing, tool/function calls, and multi-turn conversations.
• Evaluate agent outputs for correctness, relevance, groundedness, consistency, tone and completeness, using both human review and automated methods.
• Build and maintain eval datasets (golden sets, edge cases, adversarial prompts) and automated regression suites that run as agents, prompts, models and tools change.
• Identify cross-agent, orchestration and integration issues: wrong agent selection, broken handoffs, tool-call failures, context loss, looping or runaway behavior.
• Test for hallucination, prompt injection, jailbreaks, data leakage and unsafe or out-of-scope responses.
• Detect regressions caused by model upgrades, prompt changes or retrieval/knowledge changes, and quantify them with metrics.
• Triage and root-cause issues with Engineering and Product, separating prompt, model, retrieval, tool and orchestration causes.
• Re-test fixes, validate the end-user experience, and keep evals current as the product evolves.
• Document test strategy, eval results and quality trends clearly for global stakeholders. Qualifications

Qualifications

• Bachelors degree in software engineering, computer science or a related field.
• 3-5 years of QA experience, with at least 1-2 years testing LLM-based applications, AI agents or agentic workflows.
• Hands-on experience evaluating non-deterministic outputs: defining pass/fail criteria, rubrics, and acceptance thresholds for AI responses.
• Experience building or maintaining AI eval or regression suites (e.g., LLM-as-judge, golden datasets, scoring rubrics).
• Understanding of agentic concepts: tool/function calling, multi-agent orchestration, planning, memory/context, RAG.
• Experience with test automation tools (Selenium, Playwright, TestComplete or similar) and API testing.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.