Senior QA AI Testing Engineer

NewVision; Hyderabad; On-site; INR 2,500,000 - 5,000,000; Full time

Job summary

NewVision is seeking a Senior AI Test Engineer to validate the quality and governance of AI systems, including LLMs, RAG pipelines, and AI agents. You will define testing strategies and build robust evaluation datasets and benchmarks for reliable AI performance.

You will automate tests, perform red-team assessments, and report insights to guide releases. Proficiency in Python, Pytest, Java, Selenium, Playwright, and Azure OpenAI is essential, with CI/CD and GitHub Actions experience.

Qualifications

• 7–8 years in AI Test Automation
• Proven experience testing AI/ML/LLM-based systems
• Understanding of prompt behavior, model evaluation, risk management
• Hands-on with Python, Pytest, Java
• Experience with Selenium and Playwright
• Azure AI Services / Azure OpenAI, CI/CD and GitHub Actions
• DevOps integration of test automation

Responsibilities

• Define AI test strategy for LLMs, AI agents, and orchestration workflows
• Design and maintain gold evaluation datasets, prompt test cases, benchmarking frameworks
• Perform regression testing across prompts, agent logic, multi-step workflows, model upgrades
• Measure accuracy, latency, throughput, and cost of AI services
• Integrate AI test suites into CI/CD with automated quality gates
• Conduct red-team testing for safety, compliance, security and prompt-injection vulnerabilities
• Provide actionable insights through evaluation reports and release readiness assessments

Skills

Python

Pytest

Java

LLM evaluation

CI/CD

GitHub Actions

Selenium

Playwright

Azure OpenAI

Tools

Selenium

Playwright

CI/CD pipelines

Azure AI Services

Job description

Job Description:

The Senior AI Test Engineer is responsible for validating the quality, reliability, safety, performance, and governance of AI systems, including LLMs, RAG pipelines, and AI agents. Unlike traditional QA role, this position focuses on LLM evaluation, behavioral testing, hallucination, bias detection, AI test automation, and alignment with Responsible AI governance standards.

Key Responsibilities:

• Define AI test strategy for LLMs, AI agents, and orchestration workflows
• Design and maintain golden evaluation datasets, prompt test cases, and benchmarking frameworks for LLMs, RAG systems
• Perform regression testing across prompt updates, agent logic changes, multi-step agent workflows and model upgrades
• Measure and analyze accuracy, consistency, latency, throughput, and cost efficiency of AI services
• Integrate AI test suites into CI/CD pipelines with automated quality gates
• Conduct red-team testing to identify safety, compliance, security, and prompt-injection vulnerabilities
• Provide actionable insights through evaluation reports, metrics, and release readiness assessments

Qualifications:

• 7–8 years of experience in AI Test Automation
• Proven experience in testing AI/ML/LLM-based systems
• Strong understanding of:
• Prompt behavior, model evaluation, dataset curation and risk management
• Hands-on experience with:
• Python, Pytest, Java
• Test automation tools (Selenium, Playwright)
• AI evaluation tools and frameworks
• Azure AI Services / Azure OpenAI, CI/CD pipelines and GitHub Actions
• Test automation integration into DevOps workflows