Subcontractor- Gen AI QA

Birlasoft; Bengaluru; On-site; INR 3,500,000 - 6,000,000; Full time

Job summary

Birlasoft in Bengaluru is seeking an AI QA Engineer to validate and automate testing for Agentic AI and GenAI systems with strong Python expertise. You will build Python-based automation frameworks, test LLM outputs, evaluate multi-agent pipelines, and ensure robust CI/CD integration with comprehensive test coverage.

Candidates should have 7+ years in QA, with hands-on GenAI/ML exposure and API testing experience.

Qualifications

• 7+ years of QA experience, exposure to GenAI/ML systems.
• Strong Python development and test automation skills.
• Experience validating LLM outputs and multi-agent workflows.
• Familiarity with CI/CD and API testing.

Responsibilities

• Develop Python-based automation frameworks (Pytest).
• Test LLM outputs (accuracy, hallucination, relevance).
• Validate multi-agent workflows and pipelines.
• Build evaluation metrics (F1, precision, recall).
• Perform API and integration testing.
• Ensure high test coverage and CI/CD integration.

Skills

Python programming

Automation frameworks (Pytest/unittest

GenAI/LLMs understanding

API testing

CI/CD and version control

Tools

Pytest

unittest

Git

CI/CD pipelines

Job description

Job Description:

Area(s) of responsibility

AI QA Engineer – Agentic AI / GenAI

Role Overview

QA Engineer responsible for validating and automating testing for Agentic AI and GenAI systems with strong Python expertise.

Key Responsibilities

• Develop Python-based automation frameworks (Pytest)
• Test LLM outputs (accuracy, hallucination, relevance)
• Validate multi-agent workflows and pipelines
• Build evaluation metrics (F1, precision, recall)
• Perform API and integration testing
• Ensure high test coverage and CI/CD integration

Required Skills

• Strong Python programming
• Automation frameworks ( Pytest/unittest)
• Understanding of GenAI/LLMs
• API testing and debugging
• CI/CD and version control

Experience

7 years or more in QA experience, exposure to GenAI/ML systems