Automation Testing-AI

Sonata Software; Hyderabad, Chennai District, Bengaluru; Hybrid; INR 1,200,000 - 2,500,000; Full time

Job summary

Sonata Software in Hyderabad seeks an AI QA / LLM Automation Engineer to design, develop, and execute testing strategies for generative AI systems, including LLMs, RAG pipelines, and autonomous agents.

You will build Python automation, integrate tests into CI/CD pipelines, define non-deterministic evaluation metrics, and collaborate with AI researchers and product managers to improve accuracy, context retention, and reliability.

Qualifications

• Proficient in Python with clean, scalable automation scripts.
• Deep understanding of LLM architectures and prompt engineering.
• Experience testing RAG pipelines and agentic frameworks.
• Experience validating non-deterministic outputs with semantic evaluation.
• Familiar with CI/CD tools and version control.

Responsibilities

• AI Model Validation: Design and execute test strategies for evaluating LLM responses with focus on accuracy and reducing hallucinations.
• AI Agent Testing: Test the logic, state management, and tool-calling capabilities of autonomous AI agents.
• Python Automation: Develop custom automated testing frameworks using Python to support CI/CD pipelines.
• CI/CD Integration: Integrate testing frameworks into existing CI/CD pipelines for ongoing evaluation.
• Evaluation Metrics: Define metrics for non-deterministic AI features where exact-match is insufficient.
• Cross-Functional Collaboration: Work with AI developers, data scientists, and product managers to set quality benchmarks.

Skills

Python

LLM architectures

RAG pipelines

Agent frameworks

CI/CD basics

Non-deterministic testing

Problem solving

Tools

LangChain

LlamaIndex

Git

GitHub Actions

GitLab CI

Jenkins

Job description

Role & responsibilities

Job Description: AI QA / LLM Automation Engineer

Job Overview

We are seeking a highly skilled and innovative AI QA / LLM Automation Engineer to join our team. In this pivotal role, you will be responsible for designing, developing, and executing sophisticated testing strategies tailored specifically for generative AI systems. You will play a critical role in evaluating Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) pipelines, and autonomous AI agents to ensure high accuracy, contextual relevance, and robust logical reasoning.

Key Roles & Responsibilities

• AI Model Validation: Design and execute comprehensive test strategies specifically for evaluating LLM responses. Focus heavily on measuring accuracy, ensuring long-term context retention, and drastically reducing or mitigating hallucinations.
• AI Agent Testing: Rigorously test the underlying logic, complex state management, and tool-calling capabilities of autonomous AI agents across various edge cases.
• Python Automation: Develop, maintain, and scale custom automated testing frameworks from scratch using Python.
• CI/CD Integration: Continuously evaluate AI models by seamlessly integrating custom testing and evaluation frameworks into existing CI/CD pipelines.
• Evaluation Metrics: Define and implement novel evaluation metrics for non-deterministic AI features where standard deterministic software testing (exact-match) falls short.
• Cross-Functional Collaboration: Work closely with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.

Required Skills & Qualifications

• Programming: Strong programming proficiency in Python, with an emphasis on writing clean, scalable, and maintainable automation scripts.
• AI/ML Knowledge: Deep understanding of Large Language Model (LLM) architectures, prompt engineering, and model behaviors.
• Framework Experience: Hands-on experience building or testing RAG (Retrieval-Augmented Generation) pipelines and agentic frameworks (e.g., LangChain, LlamaIndex).
• Non-Deterministic Testing: Proven experience in validating non-deterministic outputs where traditional exact-match assertions do not apply (e.g., semantic similarity, LLM-as-a-judge, or custom evaluation heuristics).
• Automation & DevOps: Solid understanding of CI/CD tools (e.g., GitHub Actions, GitLab CI, Jenkins) and version control (Git).
• Problem Solving: Strong analytical mindset with the ability to foresee edge cases in generative AI applications.
• Key Responsibilities:
• AI Model Validation: Design and execute test strategies specifically for evaluating LLM responses, focusing on accuracy, context retention, and reducing hallucinations.
• AI Agent Testing: Test the logic, state management, and tool-calling capabilities of autonomous AI agents.
• Python Automation: Develop custom automated testing frameworks using Python to continuously evaluate AI models in CI/CD pipelines.
• Required Skills:
• Strong programming proficiency in Python.
• Deep understanding of LLM architectures, RAG (Retrieval-Augmented Generation), and agentic frameworks (e.g., LangChain, LlamaIndex).
• Experience validating non-deterministic outputs where traditional exact-match assertions do not apply.