AI QA Engineer

Aventra.AI

Chennai

On-site

INR 1,200,000 - 2,100,000

Full time

5 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Aventra.AI in Chennai, India seeks a Senior Generative AI QA Engineer with 8+ years of software testing and QA experience. You will lead QA for GenAI and LLM-based applications, design automation frameworks in Python, and verify AI models for accuracy and safety.

You will validate RAG pipelines, conduct prompt and security testing, and work with AI/ML teams to ensure scalability, reliability, and quality across enterprise GenAI deployments.

Qualifications

  • 8+ years of experience in Software Testing / Quality Engineering.
  • Hands-on Python-based test automation experience.
  • Experience testing Generative AI applications and LLM-based solutions.
  • Strong knowledge of LLM concepts, Prompt Engineering, and AI response evaluation.
  • API testing with Postman, REST Assured, or Python Requests.
  • Experience with RAG architecture, embeddings, vector databases, and document retrieval.
  • Understanding of hallucination detection, response validation, and contextual accuracy.
  • Experience in functional, integration, regression, and performance testing.
  • Experience designing and maintaining scalable automation frameworks.

Responsibilities

  • Lead QA strategy, test planning, and quality engineering for Generative AI and LLM-based apps.
  • Design and implement automated testing frameworks using Python.
  • Perform functional, integration, regression, API, performance, and end-to-end testing.
  • Validate LLM-generated responses for accuracy and context.
  • Test RAG pipelines including ingestion, embeddings, and retrieval accuracy.
  • Identify AI hallucinations, bias, toxicity, and unreliable responses.
  • Develop evaluation datasets, benchmarks, and CI validation pipelines.
  • Conduct prompt testing and adversarial testing across multiple models.
  • Validate AI agents, tool calls, and API integrations.
  • Evaluate LLM performance with faithfulness and context metrics.
  • Perform AI security testing including prompt injection and data leakage.
  • Execute load, stress, latency, and scalability testing for AI apps.
  • Integrate test suites into CI/CD; collaborate with teams.
  • Mentor QA engineers and promote AI testing best practices.

Skills

8+ years overall experience
Python-based test automation
Generative AI applications
LLM concepts & AI evaluation
API testing
RAG architecture
AI safety testing & bias detection
functional/integration/performance测试
automation framework design

Tools

Playwright
Selenium
Pytest
Robot Framework
OpenAI
Azure OpenAI
Anthropic Claude
Gemini
Llama
LangChain
LangGraph
CrewAI

Job description

Job Title: Senior Generative AI QA Engineer


Experience: 8+ Years


Job Summary

We are looking for an experienced Senior Generative AI QA Engineer with 8+ years of overall experience in software testing and quality engineering, including hands‑on experience in testing Generative AI applications, Large Language Models (LLMs), RAG‑based systems, and AI‑powered solutions.


The ideal candidate should have strong expertise in test automation, Python, API testing, and AI evaluation methodologies. The candidate will be responsible for ensuring the accuracy, reliability, performance, security, and scalability of enterprise GenAI applications.


Key Responsibilities


  • Lead QA strategy, test planning, and quality engineering activities for Generative AI and LLM‑based applications.

  • Design and implement automated testing frameworks using Python.

  • Perform functional, integration, regression, API, performance, and end‑to‑end testing.

  • Validate LLM‑generated responses for accuracy, relevance, consistency, and contextual understanding.

  • Test RAG pipelines, including document ingestion, chunking, embeddings, vector search, retrieval accuracy, and response generation.

  • Identify and evaluate AI hallucinations, bias, toxicity, and inconsistent model responses.

  • Develop evaluation datasets, test benchmarks, and automated validation pipelines for LLM applications.

  • Perform prompt testing, prompt regression testing, and adversarial testing across multiple LLM models.

  • Validate AI agents, tool calling, multi‑agent workflows, and API integrations.

  • Evaluate LLM performance using metrics such as faithfulness, answer relevance, context precision, and context recall.

  • Conduct AI security testing, including prompt injection, jailbreak attacks, data leakage, and sensitive information exposure.

  • Perform load, stress, latency, and scalability testing for AI applications.

  • Integrate automated test suites into CI/CD pipelines.

  • Collaborate with AI/ML engineers, developers, architects, and product teams to improve application quality.

  • Mentor QA engineers and drive best practices in AI testing and automation.


Mandatory Technical Skills


  • 8+ years of overall experience in Software Testing / Quality Engineering.

  • Strong hands‑on experience in Python‑based test automation.

  • Experience testing Generative AI applications and LLM‑based solutions.

  • Strong knowledge of LLM concepts, Prompt Engineering, and AI response evaluation.

  • Hands‑on experience with API testing using Postman, REST Assured, or Python Requests.

  • Experience with RAG architecture, embeddings, vector databases, and document retrieval.

  • Understanding of hallucination detection, response validation, and contextual accuracy.

  • Experience in functional, integration, regression, and performance testing.

  • Experience designing and maintaining scalable automation frameworks.


Preferred Technical Skills


  • Playwright, Selenium, Pytest, or Robot Framework.

  • RAGAS, DeepEval, TruLens, LangSmith, or Promptfoo.

  • OpenAI, Azure OpenAI, Anthropic Claude, Gemini, and Llama.

  • LangChain, LangGraph, CrewAI, or other Agentic AI frameworks.

  • AI security testing and OWASP Top 10 for LLM Applications.

  • Jenkins, GitHub Actions, Azure DevOps, and CI/CD.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Hiring For GenAI QA
Hiring For GenAI QA

1point1 • Navi Mumbai

On-site
INR 1,000,000 - 1,800,000
Lead AI Test Engineer
Lead AI Test Engineer

Qentelli LLC • Hyderabad

On-site
INR 1,800,000 - 2,400,000
AI QA Lead Engineer
AI QA Lead Engineer

Qentelli LLC • Hyderabad

On-site
INR 1,200,000 - 1,800,000
Senior QA Engineer, AI
Senior QA Engineer, AI

ICE • Hyderabad

On-site
INR 1,800,000 - 3,000,000
QA Engineer
QA Engineer

TigerHire • India

Remote
INR 1,200,000 - 2,000,000
Senior QA Engineer, AI
Senior QA Engineer, AI

Intercontinental Exchange Holdings, Inc. • Hyderabad

On-site
INR 2,000,000 - 4,000,000
Gen AI QA Engineer
Gen AI QA Engineer

Deqode • Bengaluru

On-site
INR 1,200,000 - 2,100,000
Senior QA AI Testing Engineer
Senior QA AI Testing Engineer

NewVision • Hyderabad

On-site
INR 2,500,000 - 5,000,000
Generative AI Engineer
Generative AI Engineer

Tekskills Inc. • Pune District

On-site
INR 1,200,000 - 1,600,000
AI Test Engineer
AI Test Engineer

Deqode • Bengaluru

On-site
INR 900,000 - 1,500,000