AI Test Engineer

AXA Global Business Services (AXA GBS); Pune District; On-site; INR 900,000 - 1,500,000; Full time

Job summary

AXA Global Business Services is seeking an experienced AI testing engineer in Pune to validate agentic AI platforms built with LLMs and orchestration frameworks. You will design test strategies for ML models, evaluate bias, explainability, and robustness, and perform adversarial testing across insurance scenarios.

Collaboration with data scientists and DevOps is essential. The role emphasizes developing automated tests, validating data pipelines, and ensuring scalable, production-ready AI

Qualifications

• Experience in software testing with focus on AI systems.
• Hands-on Python for test automation and data validation.
• Knowledge of Gen AI frameworks (LangChain/LangGraph).
• CI/CD pipelines experience (Jenkins/GitHub Actions).
• Cloud deployments for AI solutions.

Responsibilities

• Design and execute test strategies for ML models (classification, NLP, CV).
• Validate model accuracy, bias detection, explainability, and robustness.
• Perform adversarial and edge-case testing for AI systems.
• Test autonomous agents, multi-step reasoning, and state transitions.
• Validate LLM outputs for grounding, safety, and consistency.
• Evaluate prompt robustness and retrieval correctness.
• Test vector store behavior and document chunking logic.
• Validate tool integrations and fallback behaviors for failures.
• Execute regression testing for retrained models.
• Document test cases, results, and evaluation reports.
• Collaborate with data scientists and ML engineers to refine workflows.

Skills

AI testing

Python test automation

LangChain/LangGraph

CI/CD pipelines

Cloud platforms

API testing

Data validation

Security testing AI

Observability tools

Big Data testing

Tools

Docker/Kubernetes

Job description

• Experience in software testing, specializing in Artificial Intelligence (AI) systems and assure quality of an Agentic AI platform built using LLMs, LangChain/LangGraph, CrewAI, and related orchestration frameworks. The role includes testing autonomous agent behaviour, multi agent workflows, RAG pipelines, tool integrations, and domain-specific flows across insurance scenarios such as claims, underwriting, and policy servicing. The role involves validating AI/ML models, ensuring robustness of end-to-end machine learning workflows, and driving quality engineering practices across data, models, and deployment environments. The engineer will collaborate with data scientists, ML engineers, and DevOps teams to ensure trustworthy, scalable, and production-ready AI solutions.
• strong analytical skills, experience with AI workflow testing, and the ability to design robust test suites for dynamic reasoning systems.
• Strong understanding of both manual and automated testing methodologies, with the ability to design, develop, and execute test scripts.
• Responsible for the testing and delivery of parts of a product, in accordance with the customer’s requirements and interpersonal quality norms.
• Strong analytical and problem-solving skills, along with a proactive approach to identify and resolve issues, are essential.
• Responsible for ensuring that the testing process is efficient, effective, and aligned with project goals and timelines.

Responsibilities / Job Description

• Design and execute test strategies for machine learning models (classification, regression, NLP, CV, etc.).
• Validate model accuracy, fairness, bias detection, explainability, and robustness under varying datasets.
• Perform adversarial testing and edge-case validation for AI systems.
• Test autonomous agents, multi-step reasoning paths, and state transitions.
• Validate correctness, grounding, consistency, and safety of LLM outputs.
• Evaluate prompt robustness and behavioral variations across scenarios.
• Validate retrieval accuracy, grounding quality, and hallucination reduction.
• Test vector store behavior, document chunking logic, and retriever configurations.
• Validate fallback behaviors when tools or external services fail.
• Execute adversarial, prompt-injection, and red team style testing.
• Validate compliance with data privacy, business rules, and insurance guidelines.
• Benchmark LLM latency, throughput, and multi-agent performance.
• Validate concurrency handling, degradation logic, and retry mechanisms.
• Document test cases, behavior maps, prompt variations, and evaluation reports.
• Work closely with AI engineers and SMEs to refine agent workflows.

Functional & Non-Functional Testing

• Conduct API testing for AI services and model endpoints.
• Validate performance, scalability, and latency of AI inference services.
• Perform security testing for data pipelines and model-serving endpoints.

Quality Governance & Compliance

• Ensure compliance with AI ethics, data privacy, and regulatory standards.
• Document test strategies, results, and provide audit-ready evidence.
• Work closely with data scientists, ML engineers, and DevOps teams to integrate testing into the AI lifecycle.
• Mentor junior testers and contribute to building reusable test accelerators.
• Represent the testing function in client discussions, providing insights on AI quality assurance.

• Develop automated test frameworks for AI/ML pipelines.
• Implement synthetic data generation for edge-case testing.
• Conduct bias and fairness testing across diverse datasets.
• Validate monitoring dashboards for model drift and performance degradation.
• Perform regression testing for retrained models.
• Create reusable test assets for AI projects (scripts, datasets, frameworks).
• Provide detailed defect analysis and SLA-driven closure.

Required Skills

• Strong experience in software testing (manual + automation) with focus on AI systems.
• Hands-on expertise in Python for test automation, data validation, and AI testing scripts.
• Knowledge of Gen AI frameworks (Langchain, Langgraph, CrewAI).
• Familiarity with CI/CD pipelines (Jenkins, GitHub Actions, GitLab CI).
• Strong understanding of data validation, model evaluation metrics, and drift detection.
• API testing experience (REST, GraphQL) with tools like Postman or REST Assured.
• Solid knowledge of cloud platforms (AWS, Azure, GCP) for AI deployments.
• Exposure to AI ethics and compliance testing.
• Knowledge of containerization & orchestration (Docker, Kubernetes).
• Experience with Big Data ecosystems (Spark, Hadoop) for large-scale AI testing.
• Familiarity with observability tools (Prometheus, Grafana, ELK stack).
• Understanding of security testing for AI pipelines (data poisoning, adversarial attacks).