AI QA Engineer / Generative AI Test Engineer

Cybage; Pune District; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Cybage in Pune, India is seeking an AI QA Engineer to design, develop, and maintain a comprehensive testing framework for Generative AI and RAG applications. You will validate AI-generated responses, evaluate retrieval quality, and ensure the reliability and security of AI-powered solutions across engineering teams.

The role requires hands-on experience with AI evaluation tools, strong analytical skills, and exposure to Python scripting.

Qualifications

• Experience validating AI responses for accuracy and relevance.
• Knowledge of GenAI concepts including LLMs and RAG.
• Hands-on experience with AI evaluation tools such as Ragas and Giskard.
• Proficiency in prompt engineering and embeddings.
• Understanding of AI security testing, including prompt injection and data leakage.
• Strong analytical and problem-solving skills.

Responsibilities

• Design and maintain a scalable testing framework for GenAI apps.
• Create test strategies and validation plans for AI-powered solutions.
• Evaluate AI outputs for accuracy, relevance, groundedness, and citations.
• Test prompt behavior, conversation flows, and knowledge sources.
• Assess document retrieval quality and AI knowledge sources integration.
• Conduct security testing for prompt injection and data leakage.
• Collaborate with AI engineers to improve model performance.
• Define quality metrics and reporting for continuous evaluation.
• Drive best practices and continuous improvement in AI testing methodologies.

Skills

Generative AI concepts

LLMs

RAG

Prompt Engineering

Embeddings

Vector Databases

AI evaluation

Hallucination Detection

Citation Validation

Knowledge Sources

AI Integrations

Prompts

Conversation Flows

Data Leakage

Tools

Ragas

Giskard

Job description

We are looking for an AI QA Engineer to design, develop, and maintain a comprehensive testing framework for Generative AI and RAG (Retrieval-Augmented Generation) applications. The role involves validating AI-generated responses, evaluating retrieval quality, testing prompt behavior, and ensuring the reliability, security, and accuracy of AI-powered solutions developed by engineering teams.

Technical and Professional Requirements

Mandatory

• Strong understanding of Generative AI concepts, including:
  • Large Language Models (LLMs)
  • Retrieval-Augmented Generation (RAG)
  • Prompt Engineering
  • Embeddings
  • Vector Databases
• Experience validating AI responses for:
  • Accuracy
  • Relevance
  • Groundedness
  • Consistency
  • Hallucination Detection
  • Citation Validation
• Hands-on experience with AI evaluation tools such as:
  • Ragas
  • Giskard
• Experience testing:
  • Prompts
  • Conversation Flows
  • Knowledge Sources
  • Document Retrieval Mechanisms
  • AI Integrations
• Understanding of AI security testing, including:
  • Prompt Injection
  • Data Leakage
  • Unauthorized Information Disclosure
• Strong analytical and problem-solving skills.

Preferred

• Experience building automated testing frameworks for AI/LLM-based applications.
• Exposure to Python programming and automation scripting.
• Experience with vector databases such as Pinecone, Chroma, Weaviate, or FAISS.
• Understanding of AI model evaluation metrics and benchmarking methodologies.
• Experience working with cloud-based AI platforms and APIs.
• Familiarity with CI/CD pipelines and automated quality assurance practices.

Job Responsibilities

• Design and develop a scalable testing framework for validating RAG-based AI applications.
• Create test strategies, test cases, and validation approaches for AI-powered solutions.
• Evaluate AI-generated responses for accuracy, relevance, groundedness, consistency, and citation correctness.
• Validate document retrieval quality and knowledge-source effectiveness.
• Test prompts, conversation flows, and end-user interactions across multiple scenarios.
• Perform security testing focused on prompt injection, jailbreak attempts, and data leakage risks.
• Utilize tools such as Ragas and Giskard to measure and improve AI system quality.
• Collaborate with AI engineers and development teams to identify gaps and improve model performance.
• Define quality metrics, reporting mechanisms, and continuous evaluation processes for AI applications.
• Drive best practices and continuous improvement in AI testing methodologies.