AI Test Engineer -Testing Services

Birlasoft; Bengaluru; Hybrid; INR 1,200,000 - 1,800,000; Full time

Job summary

Birlasoft in Bengaluru is seeking a Senior AI Test Engineer to define and execute end-to-end test strategies for AI-enabled applications. The role focuses on Generative AI, LLM, RAG, and AI Agent workflows with API and UI considerations.

You will perform functional, integration, regression, system, and UAT testing, validate RESTful APIs, and build robust API automation frameworks while collaborating with cross-functional teams.

Qualifications

• 4–5 years of experience in AI/test engineering.
• Experience testing AI-enabled applications, including Generative AI, LLM, RAG, and AI Agent workflows.
• Strong knowledge of API testing and automation, plus DB validation using SQL.

Responsibilities

• Define and implement end-to-end test strategies for AI-enabled applications.
• Design and execute comprehensive test plans for Generative AI, LLM, RAG, and AI Agent workflows.
• Perform functional, integration, regression, system, and UAT testing.
• Validate RESTful APIs using Postman, Swagger, REST Assured, or similar frameworks.
• Develop and maintain API automation frameworks.
• Validate AI model responses for accuracy, relevance, consistency, bias, toxicity, and hallucinations.
• Test prompt engineering scenarios and optimize prompt quality.
• Validate Retrieval-Augmented Generation (RAG) pipelines and vector search accuracy.
• Verify AI guardrails, content moderation, and responsible AI compliance.
• Perform database validation using SQL.
• Conduct performance and load testing for APIs and AI services.
• Collaborate with Product Owners, Architects, Data Scientists, ML Engineers, and Developers

Skills

API testing

Test automation

SQL

Load testing

UAT testing

Tools

Postman

Swagger

REST Assured

Job description

Select how often (in days) to receive an alert:

Location: INDIA - BENGALURU - AUTOMOTIVE

Description:

Area(s) of responsibility

.Job Title: Senior AI Test Engineer
Experience: 4–5 Years
Location: Hybrid / Onsite
Employment Type: Full-time

Key Responsibilities

• Define and implement end-to-end test strategies for AI-enabled applications.
• Design and execute comprehensive test plans for Generative AI, LLM, RAG, and AI Agent workflows.
• Perform functional, integration, regression, system, and UAT testing.
• Validate RESTful APIs using tools such as Postman, Swagger, REST Assured, or similar frameworks.
• Develop and maintain API automation frameworks.
• Validate AI model responses for:
• Accuracy
• Relevance
• Consistency
• Hallucinations
• Bias
• Toxicity
• Test prompt engineering scenarios and optimize prompt quality.
• Validate Retrieval-Augmented Generation (RAG) pipelines and vector search accuracy.
• Verify AI guardrails, content moderation, and responsible AI compliance.
• Perform database validation using SQL.
• Conduct performance and load testing for APIs and AI services.
• Collaborate with Product Owners, Architects, Data Scientists, ML Engineers, and Developers