Senior Automation Architect - Python & GenAI Testing

Birlasoft Limited; Bengaluru; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Birlasoft Limited in Bengaluru is seeking a Senior Architect to drive Python-based automation testing initiatives and design scalable automation frameworks.

You will validate GenAI/LLMs interactions, build robust evaluation metrics, and ensure API and integration testing with strong CI/CD integration across teams.

Qualifications

• Strong Python programming.
• Automation frameworks (Pytest/unittest).
• Understanding of GenAI/LLMs API testing and debugging.
• CI/CD and version control.

Responsibilities

• Develop Python-based automation frameworks (Pytest).
• Test LLM outputs (accuracy, hallucination, relevance).
• Validate multi-agent workflows and pipelines.
• Build evaluation metrics (F1, precision, recall).
• Perform API and integration testing. Ensure high test coverage and CI/CD integration.

Skills

Python programming

Pytest

unittest

GenAI/LLMs testing

CI/CD

Tools

Python

Pytest

Job description

Sr Architect

Area(s) of responsibility

Key Responsibilities

• Develop Python-based automation frameworks (Pytest)
• Test LLM outputs (accuracy, hallucination, relevance)
• Validate multi-agent workflows and pipelines
• Build evaluation metrics (F1, precision, recall)
• Perform API and integration testingEnsure high test coverage and CI/CD integration

Required Skills

• Strong Python programming
• Automation frameworks (Pytest/unittest)
• Understanding of GenAI/LLMs API testing and debugging
• CI/CD and version control

Experience Level

Senior Level