AI Testing Specialist

Seven N Half; Hyderabad; On-site; INR 1,200,000 - 2,100,000; Full time

Job summary

Seven N Half in Hyderabad is seeking an AI Test Engineer to lead testing of AI/ML systems and LLM-based applications. You will develop automated evaluation and regression pipelines, focusing on accuracy, hallucinations, and security vulnerabilities.

The role requires 5–15 years of hands-on experience with AI testing, Python scripting, and test automation. You will work with LangGraph architectures and contribute to CI/CD quality gates using GitHub Actions.

Qualifications

• 5 -15 years of experience in AI testing or software QA for AI/ML systems.
• Hands-on experience with testing AI agents, LLM-based applications, and RAG systems.
• Strong scripting in Python and test automation expertise.

Responsibilities

• Develop automated testing and evaluation pipelines using Python and pytest.
• Test AI agents within LangGraph-based architectures and multi-agent setups.
• Design LLM evaluation frameworks with custom scoring and benchmarking.
• Integrate AI testing into CI/CD pipelines using GitHub Actions or similar.
• Identify hallucinations, prompt injections, and security vulnerabilities in AI systems.

Skills

Python

AI/ML testing

LLM applications testing

pytest

Tools

CI/CD

GitHub Actions

Job description

We're hiring an AI Test Engineer / AI Testing Specialist for a technology-driven organization.

Position: AI Testing Specialist
Experience: 5 -15yrs

Skills: Python Testing with experience in testing AI/ML systems or LLM-based applications

Location: Hyderabad

Work Mode: 5 Days WFO
Employment Type: Full-Time, Permanent with MNC

Job Summary

We are looking for an AI Test Engineer with hands‑on experience in testing and evaluating LLM-based applications, RAG systems, Agentic AI architectures. The role involves building automated evaluation and regression frameworks, testing AI agents for accuracy, hallucinations and security vulnerabilities, and integrating AI quality gates into CI/CD pipelines.

Key Responsibilities & Requirements-

• Develop automated testing and evaluation pipelines using Python and pytest.
• Strong understanding of test automation, evaluation pipelines, and scripting in Python.
• Exposure to AI/ML system testing, LLM application testing, adversarial testing, and automated benchmarking is required.
• Develop adversarial and red‑team testing suites to identify hallucinations, prompt injection vulnerabilities, edge‑case failures, and unexpected agent behaviour.
• Build and maintain automated regression test suites to validate AI agent consistency across model updates, prompt changes, and skill‑file modifications.
• Test AI agents within LangGraph‑based architectures, including Refinement, Decision, and Coding Agents.
• Design and implement LLM evaluation frameworks using RAGAS, custom scoring functions, and automated accuracy benchmarking to validate AI/agent outputs.
• Integrate AI‑specific testing and quality checks into CI/CD pipelines using GitHub Actions or comparable platforms.
• Demonstrate practical knowledge of LLM evaluation methodologies, RAG evaluation, and AI quality assessment.
• Have familiarity with RAG, LangChain, LangGraph, and multi‑agent orchestration concepts.