AI Quality Engineer Contract

PTW; Hyderabad; On-site; INR 900,000 - 1,400,000; Part time

Job summary

PTW in Hyderabad is seeking an AI Quality Engineer on a contract basis to test AI-driven insights and decision-making workflows.

The role focuses on evaluating LLMs and RAGsystems, validating retrieval quality, and ensuring reliability across AI products, with a 3-month engagement and extension possibility.

Qualifications

• Strong understanding of AI/ML testing and Generative AI applications.
• Hands-on testing of LLM and RAG-based systems.
• Knowledge of LLM evaluation, hallucination detection, relevance, and response quality.
• Understanding of AI agents and recommendation systems.
• Analytical and problem-solving skills.

Responsibilities

• Test and validate AI-generated insights and workflows.
• Evaluate LLM and RAG systems for accuracy and relevance.
• Validate retrieval quality and grounding in RAG systems.
• Test AI agents and autonomous workflows across scenarios.
• Collaborate with AI/ML engineers to improve quality.
• Define evaluation criteria and quality metrics.

Skills

AI testing

LLM testing

RAG evaluation

Regression testing

Problem solving

CI/CD testing

PyTest

RAGAS

LangSmith

Observability

Tools

PyTest

RAGAS

LangSmith

CI/CD tooling

Observability tools

Job description

Job Description

AI Quality Engineer

Location: Hyderabad

Work Mode: WFO (5 days a week)

Role Type: Contractual (3 months and extension would depend on project requirement and performance)

Key Responsibilities

• Test and validate AI-generated insights, recommendations, and decision-making workflows.
• Evaluate LLM and RAG systems for accuracy, relevance, consistency, factuality, and hallucinations.
• Validate retrieval quality, context relevance, grounding, and response quality in RAG systems.
• Test AI agents and autonomous workflows across functional, negative, and edge-case scenarios.
• Define AI evaluation criteria, test datasets, quality metrics, and validation processes.
• Perform regression testing for models, prompts, RAG configurations, and AI workflows.
• Collaborate with AI/ML engineers to identify issues and improve AI system quality.

Required Skills

• Strong understanding of AI/ML and Generative AI testing.
• Hands-on experience testing LLM and RAG-based applications.
• Knowledge of LLM evaluation, hallucination detection, relevance, and response quality.
• Understanding of AI agents and recommendation systems.
• Strong analytical and problem-solving skills.

Good to Have

• Experience with PyTest and automated testing frameworks.
• Experience building automated AI evaluation and regression frameworks.
• Familiarity with tools such as RAGAS, DeepEval, LangSmith, or equivalent.
• Experience with CI/CD-based test automation, performance testing, or AI guardrails.
• Familiarity with cloud platforms (AWS/Azure/GCP) and observability tools.

Success Metrics

• High accuracy, relevance, and reliability of AI outputs.
• Strong evaluation coverage across critical AI workflows.
• Early detection and reduction of hallucinations and AI regressions.
• Reduced production AI quality issues.
• Increased confidence and trust in AI-generated insights and recommendations.