Technical Lead-AI QA

Birlasoft; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Birlasoft is seeking an experienced AI QA Engineer to validate and automate testing for Agentic AI and GenAI systems. This role demands strong Python expertise and a focus on robust automation frameworks to ensure reliable LLM performance.

The candidate will test multi-agent workflows, develop evaluation metrics, and participate in CI/CD processes to maintain high-quality AI software delivery. This position is based in Bengaluru, India and requires hands-on testing of GenAI platforms.

Qualifications

• 7+ years of QA experience, with GenAI/ML exposure.
• Strong Python programming for automation.
• Experience testing GenAI/LLMs and evaluating outputs.
• API testing and debugging expertise.
• CI/CD integration and version control familiarity.

Responsibilities

• Develop Python-based automation frameworks (Pytest) to validate GenAI/LLM systems.
• Test LLM outputs for accuracy, hallucination, and relevance.
• Validate multi-agent workflows and data pipelines.
• Build evaluation metrics (e.g., F1, precision, recall) for QA measures.
• Perform API and integration testing across services.
• Ensure high test coverage and CI/CD integration.

Skills

Python programming

Automation frameworks

GenAI/LLMs

API testing

CI/CD

Version control

Tools

Pytest

unittest

Job description

Area(s) of responsibility

AI QA Engineer - Agentic AI / GenAI

Role Overview

QA Engineer responsible for validating and automating testing for Agentic AI and GenAI systems with strong Python expertise.

Key Responsibilities

• Develop Python-based automation frameworks (Pytest)
• Test LLM outputs (accuracy, hallucination, relevance)
• Validate multi-agent workflows and pipelines
• Build evaluation metrics (F1, precision, recall)
• Perform API and integration testing
• Ensure high test coverage and CI/CD integration

Required Skills

• Strong Python programming
• Automation frameworks (Pytest/unittest)
• Understanding of GenAI/LLMs
• API testing and debugging
• CI/CD and version control

Experience

7 years or more in QA experience, exposure to GenAI/ML systems