Automation Test Engineer

Sonata Software; Bengaluru; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

Sonata Software is seeking an AI QA / LLM Automation Engineer in Bengaluru to design and execute test strategies for evaluating LLM responses, focusing on accuracy and context retention. You will validate AI models, test agent logic, and build Python-based automation in CI/CD pipelines.

Experience with non-deterministic testing, RAG pipelines, and agentic frameworks like LangChain or LlamaIndex will be highly valued. Collaboration with AI developers and product managers is essential.

Qualifications

• Proficient in Python with clean, scalable automation scripts.
• Deep understanding of LLM architectures, prompt engineering, and model behavior.
• Hands-on experience with RAG pipelines and agentic frameworks.

Responsibilities

• AI Model Validation: design and execute test strategies for evaluating LLM responses focusing on accuracy and reducing hallucinations.
• AI Agent Testing: test logic, state management, and tool-calling of autonomous AI agents.
• Python Automation: develop custom automated testing frameworks in Python.

Skills

Python automation

AI/ML knowledge

RAG pipelines

LangChain

LlamaIndex

Non-deterministic testing

CI/CD & DevOps

Problem solving

AI evals & guardrails

Education

BE/ B.Tech/ Any Engineering

Tools

LangChain

LlamaIndex

RAG pipelines

Job description

Roles And Responsibilities

• Key Responsibilities:AI QA / LLM Automation Engineer
• Job Overview
• Key Roles & Responsibilities
• AI Model Validation: Design and execute comprehensive test strategies specifically for evaluating LLM responses. Focus heavily on measuring accuracy, ensuring long-term context retention, and drastically reducing or mitigating hallucinations.
• AI Agent Testing: Rigorously test the underlying logic, complex state management, and tool-calling capabilities of autonomous AI agents across various edge cases.
• Python Automation: Develop, maintain, and scale custom automated testing frameworks from scratch using Python.
• CI/CD Integration: Continuously evaluate AI models by seamlessly integrating custom testing and evaluation frameworks into existing CI/CD pipelines.
• Evaluation Metrics: Define and implement novel evaluation metrics for non-deterministic AI features where standard deterministic software testing (exact-match) falls short.
• Cross-Functional Collaboration: Work closely with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.

Required Skills & Qualifications

• Programming: Strong programming proficiency in Python, with an emphasis on writing clean, scalable, and maintainable automation scripts.
• AI/ML Knowledge: Deep understanding of Large Language Model (LLM) architectures, prompt engineering, and model behaviors.
• Framework Experience: Hands-on experience building or testing RAG (Retrieval-Augmented Generation) pipelines and agentic frameworks (e.g., LangChain, LlamaIndex).
• Non-Deterministic Testing: Proven experience in validating non-deterministic outputs where traditional exact-match assertions do not apply (e.g., semantic similarity, LLM-as-a-judge, or custom evaluation heuristics).
• Automation & DevOps: Solid understanding of CI/CD tools (e.g., GitHub Actions, GitLab CI, Jenkins) and version control (Git).
• Problem Solving: Strong analytical mindset with the ability to foresee edge cases in generative AI applications.
• Experience in AI evals, guardrails, and testing/automating AI agents, which are critical skills for the MAP team.

Key Responsibilities

• AI Model Validation: Design and execute test strategies specifically for evaluating LLM responses, focusing on accuracy, context retention, and reducing hallucinations.
• AI Agent Testing: Test the logic, state management, and tool-calling capabilities of autonomous AI agents.
• Python Automation: Develop custom automated testing frameworks using Python to continuously evaluate AI models in CI/CD pipelines.

Required Skills

• Strong programming proficiency in Python.
• Deep understanding of LLM architectures, RAG (Retrieval-Augmented Generation), and agentic frameworks (e.g., LangChain, LlamaIndex).
• Experience validating non-deterministic outputs where traditional exact-match assertions do not apply.

Qualifications & Required Skills- BE/ B.Tech/ Any Engineering

Mandatory skills-Python, DeepEval ,AI Evals ,AIGuardrails , LLM Testing ,RAG , AI Agents , Agentic AI ,LangChain , LlamaIndex

About Sonata Software

Sonata Software is an AI-first modernization engineering company that helps enterprises transform legacy systems into intelligent, scalable business platforms. Powered by its Platformation framework and Harmoni.AI platform, Sonata delivers AI-led modernization across cloud, data, AI, Dynamics, test automation, and managed services. Headquartered in Bengaluru, India, Sonata has more than $1.2 billion in revenue and 6,400+ AI engineers supporting global delivery across regions including the US, UK, India, Malaysia, Mexico, Australia DACH, and the Nordics. With deep partnerships across Microsoft, AWS, Salesforce, and Snowflake, Sonata helps Fortune 500 enterprises accelerate innovation, improve efficiency, and drive sustainable growth.