Assistant Manager-Automation testing

Vbeyond Corporation; Bengaluru; On-site; INR 800,000 - 1,200,000; Full time

Job summary

Vbeyond Corporation is seeking a Python and GenAI/AI testing specialist to build and maintain automated test frameworks for AI applications, chatbots, and LLMs. You will automate API, functional, integration, regression, and UAT tests, validate AI outputs, and work with LangSmith, LangChain, and Agentic AI frameworks.

The role integrates with GitHub and CI/CD pipelines and requires testing across AI pipelines and microservices.

Qualifications

• Develop Python + PyTest automation frameworks for AI/GenAI applications.
• Test Chatbots, GenAI apps, LLMs, RAG pipelines and AI agents.
• Automate API, functional, integration, regression and UAT testing.
• Validate AI outputs for accuracy, relevance, hallucination, consistency and response quality.
• Use LangSmith, LangChain/LangGraph or Agentic AI frameworks for testing.
• Build automated test scripts for AI pipelines and microservices.
• Integrate automation with Git/GitHub and CI/CD pipelines.
• Analyze test results, identify AI/model issues and report defects.
• Work with Azure/AWS and databases for end-to-end AI application testing.

Responsibilities

• Develop and maintain Python + PyTest automation for GenAI scenarios.
• Create and execute test plans for AI chatbots, LLMs, and AI agents.
• Automate API, functional, integration, and regression tests across pipelines.
• Validate outputs for accuracy, relevance, and consistency.
• Leverage LangSmith, LangChain, LangGraph in test strategies.
• Develop scripts for AI pipelines and microservices verification.
• Integrate tests with GitHub workflows and CI/CD, report defects.

Skills

Python

AI testing

Test automation

Tools

PyTest

LangSmith

LangChain

LangGraph

Agentic AI frameworks

Git

GitHub

CI/CD

Azure

AWS

Databases

Job description

Roles & Responsibilities Python + GenAI/AI Testing

• Develop Python + PyTest automation frameworks for AI/GenAI applications.


• Test Chatbots, GenAI apps, LLMs, RAG pipelines and AI agents.


• Automate API, functional, integration, regression and UAT testing.


• Validate AI outputs for accuracy, relevance, hallucination, consistency and response quality.


• Use tools/frameworks such as LangSmith, LangChain/LangGraph or Agentic AI frameworks for testing.


• Build automated test scripts for AI pipelines and microservices.


• Integrate automation with Git/GitHub and CI/CD pipelines.


• Analyze test results, identify AI/model issues and report defects.


• Work with Azure/AWS and databases for end-to-end AI application testing.