Senior AI Quality Assurance Engineer - Generative AI

Birlasoft Limited; Bengaluru; On-site; INR 1,600,000 - 2,200,000; Full time

Job summary

Birlasoft Limited in Bengaluru is seeking an experienced AI QA Engineer specializing in GenAI and Agentic AI. You will validate and automate tests for GenAI systems with strong Python expertise and contribute to building robust QA pipelines.

The role focuses on developing Python-based frameworks, testing LLM outputs, validating multi-agent workflows, and delivering reliable metrics while integrating with CI/CD and version control. A strong 7+ years QA background with GenAI exposure is expected.

Qualifications

• 7+ years of QA experience, incl. GenAI/ML systems.
• Strong Python automation skills using Pytest/unittest.
• Familiar with LLM validation and multi-agent workflows.
• Experience testing APIs and debugging issues.

Responsibilities

• Develop Python-based automation frameworks (Pytest).
• Test LLM outputs for accuracy, hallucinations, and relevance.
• Validate multi-agent workflows and pipelines.
• Build evaluation metrics (F1, precision, recall).
• Perform API and integration testing.
• Ensure high test coverage and CI/CD integration.

Skills

Python

Pytest

GenAI/LLMs

API testing

CI/CD

Job description

Subcontractor- Gen AI QA Area(s) of responsibility

AI QA Engineer – Agentic AI / GenAI

Role Overview

QA Engineer responsible for validating and automating testing for Agentic AI and GenAI systems with strong Python expertise.

Key Responsibilities

• Develop Python-based automation frameworks (Pytest)
• Test LLM outputs (accuracy, hallucination, relevance)
• Validate multi-agent workflows and pipelines
• Build evaluation metrics (F1, precision, recall)
• Perform API and integration testing
• Ensure high test coverage and CI/CD integration

Required Skills

• Strong Python programming
• Automation frameworks (Pytest/unittest)
• Understanding of GenAI/LLMs
• API testing and debugging
• CI/CD and version control

Experience 7 years or more in QA experience, exposure to GenAI/ML systems

Experience Level Senior Level