AI Testing

Momento Cybertech Solutions

Hyderabad, Pune, Chennai

Hybrid

INR 1,200,000 - 2,000,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Momento Cybertech Solutions in Hyderabad, India, is seeking experts in AI evaluation to advance robust AI systems. The role focuses on building evaluation strategies for LLMs, RAG and agentic components, and ensuring quality, safety and responsible AI behavior across projects.

The ideal candidate will have hands-on experience with Python, AI SDKs, and cutting-edge evaluation tools, plus familiarity with AI observability and CI/CD integration to streamline deployment pipelines.

Qualifications

  • Proficiency in Python and AI evaluation SDKs.
  • Experience with RAG and agentic system architecture.
  • Ability to design AI evaluation strategies for LLMs, RAG and agent evaluation.
  • Golden dataset curation and synthetic dataset generation experience.
  • Experience implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience using AI evaluation tools such as LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Instrumentation to enable AI Observability/ tracing using OTEL libraries.
  • Ability to integrate AI evaluation into CI/CD pipelines.
  • Knowledge of regulatory considerations for AI systems.

Responsibilities

  • Proficiency in Python & AI evaluation SDKs
  • Strong understanding of RAG & agentic system architecture
  • Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Experience in Golden dataset curation & Synthetic dataset generation
  • Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience in using AI Evaluation tools like LangSmith, Arize, LANGFUSE, Galileo, Evidently, etc.
  • Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
  • Experience in Integrating AI evaluation into CI/CD pipeline
  • Experience in Responsible AI validation
  • Knowledge of regulatory considerations for AI systems

Skills

Python
AI evaluation
RAG
Agentic system architecture
AI assurance testing
LLM evaluation
Responsible AI
AI observability

Tools

LangSmith
Arize
Langfuse
Galileo
Evidently
OTEL libraries

Job description


Role & responsibilities
  • Proficiency in Python & AI evaluation SDKs
  • Strong understanding of RAG & agentic system architecture
  • Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
  • Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
  • Experience in Golden dataset curation & Synthetic dataset generation
  • Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc.
  • Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
  • Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
  • Experience in Integrating AI evaluation into CI/CD pipeline
  • Experience in Responsible AI validation
  • Knowledge of regulatory considerations for AI systems

Preferred candidate profile
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Test Analyst - AI Assurance
Test Analyst - AI Assurance

Hexaware Technologies Limited • India

On-site
INR 4,500,000 - 7,500,000
Artificial Intelligence Engineer
Artificial Intelligence Engineer

Hexaware Technologies Limited • Pune District, Chennai, Bengaluru

On-site
INR 1,800,000 - 3,000,000
Test Analyst - AI Assurance
Test Analyst - AI Assurance

Hexaware Technologies Limited • Chennai District

On-site
INR 800,000 - 1,100,000
Artificial Intelligence Engineer
Artificial Intelligence Engineer

Hexaware Technologies Limited • Pune District, Chennai District

On-site
INR 1,200,000 - 1,800,000
AI Automation Testing
AI Automation Testing

Hexaware Technologies Limited • Chennai District

On-site
INR 1,800,000 - 2,200,000
AI Test Engineer
AI Test Engineer

AXA Global Business Services • Pune District, Bengaluru

Hybrid
INR 1,200,000 - 1,800,000
AI Test Engineer
AI Test Engineer

AXA Global Business Services (AXA GBS) • Pune District

On-site
INR 900,000 - 1,500,000
Senior QA AI Testing Engineer
Senior QA AI Testing Engineer

NewVision • Hyderabad

On-site
INR 2,500,000 - 5,000,000
Ai Automation Cgi Chennai Bengaluru
Ai Automation Cgi Chennai Bengaluru

vibehackers • Chennai District

On-site
INR 1,200,000 - 2,400,000
AI Test Engineer
AI Test Engineer

Deqode • Bengaluru

On-site
INR 900,000 - 1,500,000