AI Test Engineer

Tata Consultancy Services; Bengaluru; In loco; INR 1.200.000 - 1.800.000; Tempo pieno

Descrizione del lavoro

Tata Consultancy Services is seeking an AI Quality Assurance Specialist to evaluate AI/LLM outputs and design test scenarios for RAG-based applications. You will validate accuracy, relevance, grounding, consistency, and completeness of AI responses.

Responsibilities include collaborating with AI engineers, product owners, and domain SMEs to embed AI evaluation into CI/CD pipelines and to develop high-quality datasets for training and testing.

Competenze

• Strong understanding of AI/LLM testing concepts and AI quality assurance.

Mansioni

• Collaborate with AI engineers, product owners, and domain SMEs to understand AI behavior and risks.
• Provide actionable insights on AI quality issues, improvements, and risks.
• Contribute to AI testing best practices, standards, and reusable assets.

Conoscenze

AI/LLM testing concepts

AI quality assurance

AI response evaluation techniques

Human evaluation

Automated evaluation

LLM-as-judge

Test data / dataset creation

Responsible AI governance

RAG evaluation frameworks

RAGA (good to have)

Strumenti

RAGAS

LangSmith

Descrizione del lavoro

• Evaluate AI/LLM responses using multiple approaches including:
• Human Evaluation
• Automated Evaluation
• Validate AI outputs for accuracy, relevance, grounding, consistency, and completeness.
• Design and execute test scenarios for RAG‑based applications, ensuring response faithfulness and reduced hallucinations.

AI Evaluation Frameworks & Tooling

• Apply or integrate AI evaluation frameworks such as:
• RAGAS
• LangSmith (good to have)
• Define evaluation metrics (precision, recall, faithfulness, toxicity, bias, etc.) and analyze results.
• Collaborate with engineering teams to embed AI evaluation into CI/CD pipelines where applicable.

Responsible AI & Governance

• Ensure compliance with Responsible AI guidelines, including:
• Safety
• Transparency
• Validate system instructions, prompt templates, and guardrails.
• Support system instruction extraction, prompt testing, and jailbreak prevention strategies.

Test Data & Dataset Management

• Understand business and functional requirements to:
• Design and curate high‑quality datasets (golden datasets, adversarial datasets, edge‑case datasets).
• Prepare datasets for training validation, evaluation, and regression testing.
• Maintain dataset versioning and traceability.

Collaboration & Quality Advocacy

• Work closely with AI engineers, product owners, and domain SMEs to understand AI behavior and risks.
• Provide actionable insights on AI quality issues, improvements, and risks.
• Contribute to AI testing best practices, standards, and reusable assets.

Required Skills & Qualifications

Mandatory Skills

• Strong understanding of AI/LLM testing concepts and AI quality assurance.
• Experience in AI response evaluation techniques (human, automated, LLM‑as‑judge).
• Ability to analyze AI outputs and identify:
• Hallucinations
• Hands‑on experience in test data / dataset creation based on requirements.
• Awareness of Responsible AI principles and governance models.

Good to Have

Experience with RAG evaluation frameworks such as RAGA