AI Test Engineer - Test Lead

Fidelity International; India; Hybrid; INR 900,000 - 1,500,000; Full time

Benefits offered by this job

Comprehensive benefits package

Flexible working options

Wellbeing support

Job summary

Fidelity International is seeking an AI Test Engineer (Test Lead) to assure quality across AI-powered applications within IM Technology. You will validate LLM-based solutions, RAG systems, and AI agents, designing evaluation frameworks and automated pipelines.

Role requires hands-on testing in production-like environments, familiarity with evaluation metrics, and collaboration with cross-functional teams to ensure robust AI capabilities.

Qualifications

• Experience testing AI/GenAI in production or pre-production environments.
• Familiarity with AI quality metrics: faithfulness, relevancy, context accuracy, and hallucination detection.
• Experience building automated AI evaluation pipelines and integrating with CI/CD workflows.
• Knowledge of vector databases and RAG architectures and how they affect testing outcomes.

Responsibilities

• Validate LLM-based applications for accuracy, consistency, and robustness across business scenarios.
• Test RAG systems evaluating retrieval quality, grounding, and answer completeness.
• Evaluate AI Agents and agentic workflows including planning and decision-making.
• Develop evaluation frameworks using DeepEval, LangSmith, Phoenix, or similar platforms.
• Create and maintain benchmark datasets and synthetic test data for continuous quality assessment.
• Collaborate with Product Owners and AI Engineers to validate AI-enabled features.

Skills

LLMs knowledge

GenAI testing

AI evaluation frameworks

CI/CD automation

Python testing

Prompt engineering

Tools

DeepEval

RAGAS

LangSmith

Phoenix

Promptfoo

TruLens

LangGraph

CrewAI

AutoGen

Semantic Kernel

Job description

About the Opportunity

Job Type: Permanent

Application Deadline: 13 September 2026

Job Description Title: AI Test Engineer (Test Lead)

Department: ISS DELIVERY - DEVELOPMENT - GURGAON

Location: INB905E

Level: 4

We’re proud to have been helping our clients build better financial futures for over 50 years. How have we achieved this? By working together - and supporting each other - all over the world. So, join our Asset Managemrtn Platform (AMP) team and feel like you’re part of something bigger.

About your team

The Investment Solutions Services (ISS) delivery team provides team provides systems development, implementation and support services for FIL’s global Investment Management businesses across asset management lifecyle. We support Fund Managers, Research Analysts, Traders and Investment Services Operations in all of FIL’s international locations, including London, Hong Kong, and Tokyo

About your role

You will join the QE team as an AI Test Engineer (Test Lead) and play a key role in assuring the quality, reliability, and trustworthiness of AI-powered applications across IM Technology. In addition to traditional software testing, you will be responsible for validating Generative AI solutions, Retrieval Augmented Generation (RAG) systems, AI agents, and Large Language Model (LLM) based applications deployed within the organisation.

Your responsibilities will include:

• Validate LLM-based applications for accuracy, consistency, robustness, and reliability across a wide range of business scenarios.
• Test RAG (Retrieval Augmented Generation) systems by evaluating retrieval quality, context relevance, grounding, answer correctness, and response completeness.
• Evaluate AI Agents and Agentic workflows including tool usage, planning, reasoning, task completion, decision-making, and failure recovery scenarios.
• Design and execute evaluation frameworks for GenAI applications using industry-standard tools such as DeepEval, RAGAS, LangSmith, Phoenix, or similar evaluation platforms.
• Assess AI systems for hallucinations, factual correctness, context adherence, citation quality, and response relevance.
• Test conversational AI systems for multi-turn interactions, context retention, memory handling, and user experience consistency.
• Validate AI guardrails, prompt templates, safety controls, content filtering mechanisms, and responsible AI requirements.
• Perform adversarial, negative, and edge-case testing to identify vulnerabilities such as prompt injection, jailbreak attempts, data leakage, and unsafe outputs.
• Create and maintain benchmark datasets, golden test sets, synthetic test data, and evaluation pipelines for continuous AI quality assessment.
• Validate model performance across different prompts, model versions, configurations, and environments.
• Understand business requirements, user stories, and AI use cases to define effective test strategies and quality gates.
• Collaborate with Product Owners, Business Analysts, AI Engineers, and Developers to validate AI-enabled features and solutions.
• Develop and maintain automation frameworks and test suites using Python and modern open-source testing tools.
• Design and execute functional, integration, regression, and end-to-end tests across traditional and AI-powered applications.

About You

• Good understanding of Large Language Models (LLMs), Generative AI concepts, prompt engineering, embeddings, vector databases, and RAG architectures.
• Hands-on experience testing AI/GenAI applications in production or pre-production environments.
• Experience with AI evaluation frameworks such as DeepEval, RAGAS, LangSmith, Phoenix, Promptfoo, TruLens, or similar tools.
• Understanding of AI quality metrics including faithfulness, answer relevancy, context precision, context recall, hallucination detection, and toxicity assessment.
• Experience testing AI Agents and agentic frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar platforms.
• Familiarity with vector databases such as Pinecone, Weaviate, Chroma, OpenSearch, Azure AI Search, or similar technologies.
• Knowledge of AI observability and monitoring concepts.
• Understanding of AI security risks including prompt injection, jailbreaks, data leakage, and model misuse scenarios.
• Experience building automated AI evaluation pipelines integrated with CI/CD processes.
• Exposure to cloud AI services such as Azure OpenAI, AWS Bedrock, Google Vertex AI, or OpenAI APIs would be advantageous.

Benefits

• Feel rewarded For starters, we’ll offer you a comprehensive benefits package.
• We’ll value your wellbeing and support your development.
• And we’ll be as flexible as we can about where and when you work – finding a balance that works for all of us.

It’s all part of our commitment to making you feel motivated by the work you do and happy to be part of our team.

For more about our work, our approach to dynamic working and how you could build your future here, visit careers.fidelityinternational.com.

For more about our work, our approach to dynamic working and how you could build your future here, visit careers.fidelityinternational.com.