Test Analyst - AI Assurance

Hexaware Technologies Limited; India; On-site; INR 1,000,000 - 2,000,000; Full time

Job summary

Hexaware Technologies Limited is seeking an AI Quality/GenAI specialist to design and evaluate GenAI systems, focusing on RAG- and agent-based architectures. The role emphasizes Python-based evaluation, observed with leading tools to ensure high-quality AI deliveries.

The candidate will curate datasets, generate synthetic data, and establish observability to trace AI behavior end-to-end, using OTEL libraries and related tooling.

Qualifications

• Strong knowledge of GenAI system architecture including RAG and agents.
• Expertise in Python for AI evaluation frameworks.
• Experience evaluating models and agents with leading tools.
• Experience with Responsible AI validation and reporting.
• Dataset curation and synthetic data generation experience.
• Ability to instrument AI observability and tracing using OTEL libraries.

Responsibilities

• Strong understanding of GenAI system architecture (RAG / Agents).
• Implement Python-based AI evaluation frameworks like Ragas, DeepEval, etc.
• Perform model and agent quality evaluation using tools like Arize, LangFuse, ConfidentAI, etc.
• Perform Responsible AI validation and report RAI metrics.
• Golden dataset curation and synthetic dataset generation.
• Instrument and establish AI observability/tracing using OTEL libraries, expert in Python.

Skills

GenAI architecture

Python

AI evaluation

Tools

Arize

LangFuse

ConfidentAI

OTEL

Job description

• Strong understanding of GenAI system architecture (RAG / Agents)
• Implement python-based AI Evaluation frameworks like Ragas, DeepEval, etc
• Perform Model & Agent quality evaluation using tools like Arize, LangFuse, ConfidentAI, etc
• Perform Responsible AI validation & report RAI metrics
• Golden dataset curation & Synthetic dataset generation
• Ability to instrument & establish AI Observability / Tracing using OTEL libraries Expert in Python