Test Analyst - AI Assurance

Hexaware Technologies Limited

India

Presencial

INR 4 500 000 - 7 500 000

Tempo integral

há 17 horas
Torna-te num dos primeiros candidatos

Uma candidatura completa num minuto

Currículo e carta de apresentação personalizados, prontos a enviar.

Ultrapassa os filtros ATS

Resumo da oferta

Hexaware Technologies Limited in India seeks an AI Assurance Specialist to lead GenAI quality, safety and responsible AI validation. You will shape evaluation strategies for LLMs, RAG and agent systems, and drive observability with OTEL.

You will curate golden and synthetic datasets, implement evaluation frameworks (Ragas, DeepEval) and integrate AI testing into CI/CD to ensure reliable, auditable AI systems.

Qualificações

  • Proven experience in AI assurance testing and QA of GenAI systems.
  • Experience with RAG and agent architectures.
  • Proficiency in Python and AI evaluation SDKs.
  • Experience with AI observability and OTEL.
  • Familiarity with responsible AI validation and regulatory considerations.

Responsabilidades

  • Develop AI Assurance test strategy for quality, safety, reliability and RAI.
  • Design evaluation strategy for LLMs, RAG and agents across dimensions.
  • Curate Golden dataset and Synthetic dataset generation.
  • Implement AI evaluation frameworks like Ragas, DeepEval.
  • Use AI evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently.
  • Instrument AI observability/tracing using OTEL libraries.
  • Integrate AI evaluation into CI/CD pipeline.
  • Perform Responsible AI validation.

Conhecimentos

GenAI architecture
Python
AI evaluation SDKs
Model Evaluation
Agent Evaluation
Responsible AI validation
RAI metrics reporting
Golden dataset curation
Synthetic data generation
AI Observability
OTEL

Ferramentas

Ragas
DeepEval
LangSmith
Arize
Langfuse
Galileo
Evidently
OpenTelemetry (OTEL)

Descrição da oferta de emprego

Role Overview

Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems.

Key Responsibilities
  • Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Experience in Golden dataset curation & Synthetic dataset generation
  • Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
  • Experience in Integrating AI evaluation into CI/CD pipeline
  • Experience in Responsible AI validation
Required Skills and Capabilities
  • Strong understanding of GenAI System Architecture of RAG & Agents
  • Proficiency in Python & AI evaluation SDKs
  • AI Evaluation Frameworks (Ragas, DeepEval)
  • Model Evaluation
  • Agent Evaluation
  • Responsible AI (RAI) Validation
  • RAI Metrics Reporting
  • Golden Dataset Curation
  • Synthetic Dataset Generation
  • AI Observability
  • AI Tracing
  • OpenTelemetry (OTEL)
  • LLM Evaluation
  • AI Quality Engineering
  • AI Governance
  • AI Monitoring
  • AI Testing & Assurance
  • Root Cause Analysis (AI Systems)
  • Knowledge of regulatory considerations for AI systems
Role-Specific Requirements
  • Proficiency in Python & AI evaluation SDKs
  • Strong understanding of RAG & agentic system architecture
  • Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Experience in Golden dataset curation & Synthetic dataset generation
  • Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
  • Experience in Integrating AI evaluation into CI/CD pipeline
  • Experience in Responsible AI validation
  • Knowledge of regulatory considerations for AI systems
Experience Requirements

More than 4 years of experience in AI Assurance Testing

Required Skills
  • DeepEval
  • LangFuse
  • Model and Agent quality evaluation
  • OTEL libraries
  • Python
  • Ragas
  • Responsible AI validation and report RAI metrics
Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

AI Testing
AI Testing

Momento Cybertech Solutions • Hyderabad, Pune, Chennai

Híbrido
INR 1 200 000 - 2 000 000
Test Analyst - AI Assurance
Test Analyst - AI Assurance

Hexaware Technologies Limited • Chennai District

Presencial
INR 800 000 - 1 100 000
AI Test Engineer
AI Test Engineer

AXA Global Business Services (AXA GBS) • Pune District

Presencial
INR 900 000 - 1 500 000
Artificial Intelligence Engineer
Artificial Intelligence Engineer

Hexaware Technologies Limited • Pune District, Chennai, Bengaluru

Presencial
INR 1 800 000 - 3 000 000
AI Testing
AI Testing

LTM • Chennai

Híbrido
INR 1 500 000 - 2 100 000
Senior Software Quality Engineer - AI Assurance
Senior Software Quality Engineer - AI Assurance

Sysvine Technologies Private Limited. • Chennai

Presencial
INR 2 000 000 - 3 500 000
Senior QA AI Testing Engineer
Senior QA AI Testing Engineer

NewVision • Hyderabad

Presencial
INR 2 500 000 - 5 000 000
AI Test Engineer
AI Test Engineer

AXA Global Business Services • Pune District, Bengaluru

Híbrido
INR 1 200 000 - 1 800 000
AI Trust and Governance Architect
AI Trust and Governance Architect

Infosys • Bengaluru

Presencial
INR 1 600 000 - 2 400 000
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Mployee.me • Mumbai

Presencial
INR 1 200 000 - 1 800 000