Sr Architect

Birlasoft; Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Birlasoft in Bengaluru is seeking a QA Automation Engineer to join an AI/GenAI testing initiative. You will develop Python-based automation frameworks using Pytest, validate LLM outputs for accuracy and relevance, and verify multi-agent workflows and pipelines.

The role includes building evaluation metrics and ensuring robust API and integration tests within CI/CD pipelines. Ideal candidates have strong Python skills, experience with Pytest/unittest, familiarity with GenAI/LLMs, and proficiency

Qualifications

• Strong Python programming skills and automation framework experience.
• Experience with Pytest and/or Unittest.
• Understanding of GenAI and large language models.
• Experience in API testing and debugging.
• Familiarity with CI/CD and version control systems.

Responsibilities

• Develop Python-based automation frameworks (Pytest).
• Test LLM outputs (accuracy, hallucination, relevance).
• Validate multi-agent workflows and pipelines.
• Build evaluation metrics (F1, precision, recall).
• Perform API and integration testing.
• Ensure high test coverage and CI/CD integration.

Skills

Python programming

Pytest/Unittest

GenAI/LLMs

API testing

CI/CD & Git

Job description

Job Description:

Area(s) of responsibility

Key Responsibilities

• Develop Python-based automation frameworks (Pytest)


• Test LLM outputs (accuracy, hallucination, relevance)


• Validate multi-agent workflows and pipelines


• Build evaluation metrics (F1, precision, recall)


• Perform API and integration testing


• Ensure high test coverage and CI/CD integration

Required Skills

• Strong Python programming


• Automation frameworks (Pytest/unittest)


• Understanding of GenAI/LLMs


• API testing and debugging


• CI/CD and version control

Requirements: