AI QA Engineer

Grey Chain; United States; Remote; USD 90,000 - 150,000; Full time

Job summary

Grey Chain is seeking an Agentic AI QA Engineer to join our team on a cutting-edge Agentic AI project. You will validate AI agents and agentic workflows, ensuring reliability, safety, and performance before production.

The role requires hands-on testing of AI agents, multi-step autonomous workflows, and collaboration with AI/ML engineers to design comprehensive test coverage. Experience with Python-based test frameworks, API testing, and CI/CD pipelines is highly valued.

Qualifications

• 3+ years of QA/testing experience, with hands-on AI agent testing.
• Knowledge of agentic AI concepts and how agents interact with tools.
• Experience with functional, integration, and end-to-end testing.
• Proficiency with Python-based test frameworks, API testing tools, and CI/CD pipelines.
• Ability to design test strategies for non-deterministic AI systems.
• Strong analytical and collaboration skills.

Responsibilities

• Design and execute test strategies and test cases tailored to agentic AI systems and multi-step autonomous workflows.
• Perform functional, integration, and end-to-end testing of AI agents across various use cases and environments.
• Identify and document edge cases, failure scenarios, and hallucination risks in AI agent behaviour.
• Validate agent decision-making, tool usage, task completion, and multi-turn interaction flows.
• Collaborate with AI/ML engineers and product teams to understand agent architecture and design comprehensive test coverage.
• Build and maintain automated test suites using relevant testing frameworks and tools.
• Analyse test results, log defects, and drive issues to closure with development teams.
• Contribute to continuous improvement of QA processes for agentic AI systems.

Skills

QA testing

Agentic AI

End-to-end testing

Python testing

CI/CD

Test strategy design

Edge cases

Tools

LangChain

AutoGen

CrewAI

Job description

Agentic AI QA Engineer

Experience: 3+ Years

Location: Delhi NCR / Remote

Notice Period: Immediate Joiners Preferred

About the Role

We are looking for a skilled Agentic AI QA Engineer to join our team working on a cutting-edge Agentic AI project. The ideal candidate will have hands-on experience testing AI agents and agentic workflows, with a strong grasp of how these systems are designed, built, and expected to behave in real-world scenarios. This role is critical to ensuring the reliability, safety, and performance of AI-driven agent systems before they reach production.

Key Responsibilities

• Design and execute test strategies and test cases tailored specifically to agentic AI systems and multi-step autonomous workflows
• Perform functional, integration, and end-to-end testing of AI agents across various use cases and environments
• Identify and document edge cases, failure scenarios, and hallucination risks in AI agent behaviour
• Validate agent decision-making, tool usage, task completion, and multi-turn interaction flows
• Collaborate closely with AI/ML engineers and product teams to understand agent architecture and design comprehensive test coverage
• Build and maintain automated test suites using relevant testing frameworks and tools
• Analyse test results, log defects, and drive issues to closure with development teams
• Contribute to continuous improvement of QA processes for agentic AI systems

Required Skills & Qualifications

• 3+ years of overall QA/testing experience, with hands-on exposure to testing AI agents or agentic workflows
• Strong understanding of Agentic AI concepts how AI agents are architected, function, and interact with tools/environments
• Practical experience with functional, integration, and end-to-end testing methodologies
• Familiarity with testing frameworks and automation tools relevant to AI systems (e.g., Python-based test frameworks, API testing tools, CI/CD pipelines)
• Ability to independently design test strategies and detailed test cases for complex, non-deterministic AI systems
• Strong analytical skills to identify edge cases, failure modes, and hallucination risks in LLM/agent outputs
• Good understanding of LLMs, prompt engineering, and prompt-based interactions is a strong plus
• Excellent problem-solving skills and attention to detail
• Strong communication skills to collaborate with cross-functional AI/engineering teams

Good to Have

• Exposure to tools like LangChain, AutoGen, CrewAI, or similar agentic frameworks
• Experience with LLM evaluation frameworks or prompt testing tools
• Prior experience in AI/ML product testing or QA for conversational AI systems