AIML Test engineer

Mphasis; Mumbai; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Mphasis Mumbai is seeking a senior QA/SDET specialized in AI testing to evaluate agent decision‑making, audit AI‑generated tests, and supervise agentic testing systems. You will validate orchestration and guardrails, design non‑deterministic test cases, and ensure reliability of multi‑agent AI workflows across Postman, MongoDB, and vector DBs.

Strong Python and CI/CD skills required. You will collaborate with AI/ML and development teams to improve agent reliability and reduce hallucinations,

Qualifications

• Strong knowledge of Software Testing / QA fundamentals.
• Experience with Manual Testing and API Testing.
• Understanding of Generative AI, LLMs, prompts, tokens, context, and hallucinations.
• Understanding of Agentic AI concepts and agent workflows.
• Knowledge of AI agents, tool/function calling, planning, execution, and memory.
• Basic understanding of RAG and vector databases.
• Ability to design test cases for non-deterministic AI outputs.
• Strong analytical, debugging, and problem-solving skills.
• Perform database testing and backend data validation using MongoDB/PostgreSQL.
• Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector.
• Python knowledge and test automation experience.
• Selenium, Playwright, PyTest, or similar automation tools.
• Knowledge of AI evaluation techniques such as groundedness, relevance, faithfulness, and response quality.
• Experience with Azure/AWS/GCP and AI services.
• Git, CI/CD, and Docker knowledge.
• Experience testing multi-agent or AI orchestration systems.
• Understanding of agentic orchestration frameworks
• Ability to evaluate AI generated code, tests, and decisions.
• Risk based thinking and judgment-heavy evaluation.

Responsibilities

• Evaluate agent decision‑making: assess whether an agent’s scope decisions, planning steps, and tool‑calling sequences are correct and safe.
• Audit AI‑generated tests: review assertions and test cases produced by agentic testing systems to ensure they verify the right behaviours.
• Supervise agentic testing systems: tune and correct what the AI testing system chooses to cover, rather than writing every test manually.
• Verify orchestration reliability: validate the orchestration framework (e.g., Agents SDK) and ensure tool/function‑calling reliability.
• Check guardrails and autonomy levels: test how much autonomy agents have in production and whether guardrails prevent unsafe actions.
• Build independent verification layers: maintain human‑owned checks that validate AI‑generated outputs.
• Test Agentic AI applications and AI-Driven Workflows: ensure end‑to‑end reliability.
• Validate agent workflows complete flow from start to end: confirm seamless operation across stages.
• Test LLM-based applications for accuracy, relevance, consistency, and hallucinations.
• Test multi-agent workflows and communication between agents.
• Test RAG (Retrieval-Augmented Generation) workflows and knowledge retrieval.
• Validate agent memory and context handling.
• Perform API testing using Postman or similar tools.
• Perform database testing and backend data validation using MongoDB/PostgreSQL.
• Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector.
• Perform basic prompt injection and AI security/guardrail testing.
• Analyse agent execution logs and identify the root cause of failures.
• Create and maintain test cases, test scenarios, bug reports, and QA documentation.
• Work closely with AI/ML and development teams to improve overall agent reliability.

Skills

QA fundamentals

API testing

Generative AI concepts

Agentic AI testing

LLM testing

Prompt testing

Memory handling

Risk-based evaluation

Multi-agent testing

Test automation

Python

Selenium

Playwright

PyTest

DevOps tooling

Tools

Postman

MongoDB

PostgreSQL

Chroma DB

PGVector

Python

Selenium

Playwright

PyTest

Docker

Git

CI/CD

Azure

AWS

GCP

Job description

• Evaluate agent decision‑making: Assess whether an agent’s scope decisions, planning steps, and tool‑calling sequences are correct and safe
• Strong Playwright experience and understanding
• Agentic AI experience and understanding
• API
• Audit AI‑generated tests: Review assertions and test cases produced by agentic testing systems to ensure they verify the right behaviours.
• Supervise agentic testing systems: Tune and correct what the AI testing system chooses to cover, rather than writing every test manually
• Verify orchestration reliability: Validate the orchestration framework (e.g., Agents SDK) and ensure tool/function‑calling reliability.
• Check guardrails and autonomy levels: Test how much autonomy agents have in production and whether guardrails prevent unsafe actions
• Build independent verification layers: Maintain human‑owned checks that validate AI‑generated outputs.
• 6+ Years of Total IT experience in QA/SDET and Testing, with 1to 2 years in the field of AI/ML domain.
• Test Agentic AI applications and AI-Driven Workflows
• Validate agent workflows complete flow from start to end
• Test LLM-based applications for accuracy, relevance, consistency, and hallucinations.
• Test multi-agent workflows and communication between agents.
• Test RAG (Retrieval-Augmented Generation) workflows and knowledge retrieval.
• Validate agent memory and context handling.
• Perform API testing using Postman or similar tools.
• Perform database testing and backend data validation using MongoDB/PostgreSQL.
• Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector
• Perform basic prompt injection and AI security/guardrail testing.
• Analyse agent execution logs and identify the root cause of failures.
• Create and maintain test cases, test scenarios, bug reports, and QA documentation.
• Work closely with AI/ML and development teams to improve overall agent reliability.

Required Skills

• Strong knowledge of Software Testing / QA fundamentals.
• Experience with Manual Testing and API Testing.
• Understanding of Generative AI, LLMs, prompts, tokens, context, and hallucinations.
• Understanding of Agentic AI concepts and agent workflows.
• Knowledge of AI agents, tool/function calling, planning, execution, and memory.
• Basic understanding of RAG and vector databases.
• Ability to design test cases for non-deterministic AI outputs.
• Strong analytical, debugging, and problem-solving skills.
• Perform database testing and backend data validation using MongoDB/PostgreSQL.
• Knowledge on Key Value Caching, Vector Databases like Chroma DB/PGVector.
• Python knowledge and test automation experience.
• Selenium, Playwright, PyTest, or similar automation tools.
• Knowledge of AI evaluation techniques such as groundedness, relevance, faithfulness, and response quality.
• Experience with Azure/AWS/GCP and AI services.
• Git, CI/CD, and Docker knowledge.
• Experience testing multi-agent or AI orchestration systems.
• Understanding of agentic orchestration frameworks
• Ability to evaluate AI generated code, tests, and decisions
• Risk based thinking and judgment-heavy evaluation

Key Skills

6+ years as QA/SDET, 2+ years in the field of AI/ML, Agentic AI Testing, Generative AI Testing, LLM Testing, RAG, Prompt Testing, AI Testing, Python, Test Automation, Use case generation and Management. Strong QA and Testing Skills, PGVector, Chroma DB, Mongo DB, PostgreSQL