GenAI QA / AI Test Engineer

Halcer; Bengaluru; On-site; INR 1,800,000 - 2,700,000; Full time

Job summary

Halcer is seeking an experienced Gen AI QA Engineer to lead the QA lifecycle for next-generation generative AI and agentic AI systems in Bengaluru. You will ensure reliability, safety, and performance across the SDLC, bridging traditional automation with non-deterministic AI validation to safeguard outputs from bias and hallucinations.

You will design end-to-end test strategies, validate prompts and model changes, and integrate automated quality gates into CI/CD, working closely with AI/ML

Qualifications

• 5 to 8 years of hands-on software testing experience spanning manual and automated testing.
• Proven track record evaluating LLM-powered applications and AI safety boundaries.
• Experience with UI and API testing using modern frameworks.

Responsibilities

• AI Test Strategy & Execution: design and maintain end-to-end test strategies for GenAI, RAG architectures, and agentic workflows.
• LLM & Agentic Output Validation: evaluate model responses to detect hallucinations, bias, toxicity and alignment issues.
• Prompt & Model Regression Testing: assess prompt iterations, model fine-tuning impact, and version changes to prevent regressions.
• Automation & CI/CD Integration: build scalable test automation suites and integrate AI quality gates into CI/CD pipelines.
• API & Integration Testing: validate end-to-end data flows, tool bindings, and REST/gRPC interfaces.
• Dataset Management: curate and maintain golden evaluation datasets for precision, recall, and safety metrics.
• Cross-Functional Collaboration: partner with AI/ML developers, prompt engineers, and product managers.

Skills

GenAI / LLM testing

Python

JavaScript/TypeScript

QA Experience

Tools

Playwright

Selenium

Postman

REST Assured

PyTest

Jenkins

GitHub Actions

Azure DevOps

GitLab CI

Job description

We are seeking an experienced Gen AI QA Engineer to lead the quality assurance lifecycle for next-generation generative AI and agentic AI systems. In this role, you will be responsible for ensuring the reliability, safety, functional accuracy, and performance of large language model (LLM) applications across the entire Software Development Life Cycle (SDLC). You will bridge traditional automation and non-deterministic AI validation, building robust test pipelines that safeguard model outputs against bias, hallucinations, and safety violations.

Job Title: Gen AI QA Engineer

Location: Bangalore, India (On-site)

Experience Level: 5–8 Years

Role Summary

We are seeking an experienced Gen AI QA Engineer to lead the quality assurance lifecycle for next-generation generative AI and agentic AI systems. In this role, you will be responsible for ensuring the reliability, safety, functional accuracy, and performance of large language model (LLM) applications across the entire Software Development Life Cycle (SDLC). You will bridge traditional automation and non-deterministic AI validation, building robust test pipelines that safeguard model outputs against bias, hallucinations, and safety violations.

Key Responsibilities

• AI Test Strategy & Execution: Design, implement, and maintain end-to-end test strategies tailored to GenAI, Retrieval-Augmented Generation (RAG) architectures, and agentic workflows.
• LLM & Agentic Output Validation: Perform rigorous evaluation of model responses to detect and mitigate hallucinations, bias, toxicity, alignment issues, and non-deterministic drift.
• Prompt & Model Regression Testing: Systematically evaluate prompt iterations, model fine-tuning impact, and version changes to prevent regression across software releases.
• Automation & CI/CD Integration: Build and maintain scalable test automation suites using modern frameworks (Playwright, Selenium) and integrate automated AI quality gates directly into CI/CD pipelines.
• API & Integration Testing: Validate end-to-end data flows, tool bindings, and REST/gRPC API interfaces connecting front-end applications, orchestrators (e.g., LangChain/LlamaIndex), and backend AI services.
• Dataset Management: Curate, synthetic-generate, and maintain golden evaluation datasets to measure baseline AI precision, recall, and safety metrics across varied user edge cases.
• Cross-Functional Collaboration: Partner closely with AI/ML developers, prompt engineers, and product managers to define clear acceptance criteria, benchmark safety thresholds, and enforce quality gates.

Required Experience & Qualifications

• Core QA Expertise: 5 to 8 years of hands-on software testing experience spanning both manual exploratory techniques and automated testing paradigms.
• GenAI / LLM Testing: Proven track record of evaluating LLM-powered applications, prompt behavior, non-deterministic outputs, and responsible AI safety boundaries.
• Automation Frameworks: Proficient in automated UI and workflow testing using Playwright, Selenium, or modern JavaScript/Python-based automation frameworks.
• API Testing: Demonstrated experience testing microservices and APIs using tools like Postman, REST Assured, or PyTest.
• CI/CD & DevOps: Practical experience embedding automated quality gates into CI/CD workflows (Jenkins, GitHub Actions, Azure DevOps, or GitLab CI).
• Scripting Skills: Strong coding ability in Python or JavaScript/TypeScript for writing test automation scripts and data validation routines.

Preferred / Nice-to-Have Skills

• Familiarity with specialized LLM evaluation frameworks (e.g., Ragas, DeepEval, TruLens, or Promptfoo).
• Hands-on exposure to RAG pipeline evaluation metrics (Faithfulness, Answer Relevance, Context Recall).
• Basic understanding of security testing for LLMs (e.g., prompt injection prevention, jailbreak testing).