QA AI Evaluator (AI testing)

BT Group

Bengaluru

Hybrid

INR 1,500,000 - 2,400,000

Full time

9 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

BT Group is seeking a QA AI Evaluator in Bengaluru to assure quality, reliability, and governance of Agentic AI solutions. You will evaluate AI agents, prompts, LLM outputs, and end-to-end workflows to ensure accurate, trusted outcomes.

Responsibilities include defining evaluation frameworks, validating AI accuracy and reasoning, benchmarking models, and presenting insights to leadership. Collaboration with Product, Engineering, and QA teams is essential.

Qualifications

  • 3-4 years in AI testing or AI/LLM evaluation roles.
  • Strong understanding of LLMs, prompt engineering, and multi-agent systems.
  • Ability to define evaluation frameworks, KPIs, scorecards, and success criteria.
  • Hands-on experience validating AI accuracy, reasoning, reliability, and business outcomes.
  • Familiarity with Responsible AI, security, privacy, governance, risk and compliance.

Responsibilities

  • Define and govern AI quality standards, evaluation frameworks, KPIs, and success criteria.
  • Validate Agentic AI solutions for accuracy, reasoning, reliability, and business effectiveness.
  • Assess prompts, LLM outputs, and end-to-end agent workflows to ensure quality outcomes.
  • Benchmark models, prompts, tools, and agent configurations to optimize performance.
  • Measure and improve AI quality metrics including accuracy, relevance, completeness, and hallucination rates.
  • Identify risks and opportunities for continuous improvement.
  • Support AI adoption, user acceptance, and organizational change initiatives.
  • Collaborate with Product, Engineering, Architecture, QA, and business stakeholders.
  • Present insights, risks, performance metrics, and strategic recommendations to leadership.

Skills

AI Testing
Quality Engineering
AI/LLM Evaluation
Agentic AI Assurance
Prompt Engineering
Evaluation frameworks
KPIs & scorecards
Root cause analysis
Data-driven QA
Stakeholder management

Tools

Evaluation tools
Test automation frameworks

Job description

About the role

The QA AI Evaluator will be responsible for assuring the quality, reliability, governance, and business effectiveness of Agentic AI solutions. The role focuses on evaluating AI agents, prompts, LLM outputs, and end-to-end workflows to ensure accurate, consistent, and trusted outcomes.

  • Define AI evaluation frameworks, standards, KPIs, and success criteria.
  • Validate agent accuracy, reasoning, reliability, workflow quality, and business outcomes.
  • Assess prompts, LLM responses, hallucination risks, and AI-generated content quality.
  • Benchmark models, tools, prompts, and agent configurations to improve performance.
  • Ensure Responsible AI, security, privacy, governance, and compliance requirements are met.
  • Provide clear insights, risks, recommendations, and scorecards to leadership and stakeholders.
What you'll need to succeed (Skills & Experience)
  • Strong experience in AI Testing, Quality Engineering, AI/LLM Evaluation, or Agentic AI Assurance.
  • Deep understanding of LLMs, Prompt Engineering, RAG, AI Agents, and Multi-Agent Architectures.
  • Expertise in defining evaluation frameworks, quality metrics, KPIs, scorecards, and benchmarking approaches.
  • Experience validating AI accuracy, reasoning, reliability, hallucinations, and business outcomes.
  • Knowledge of Responsible AI, Security, Privacy, Governance, Risk, and Compliance practices.
  • Strong analytical and problem-solving skills with experience in root cause analysis and continuous improvement.
  • Familiarity with AI evaluation tools, test automation, and data-driven quality assessment techniques.
  • Ability to collaborate effectively with Product, Engineering, Architecture, QA, and Business stakeholders.
  • Excellent communication and stakeholder management skills, including presenting insights to leadership.
  • Experience driving AI adoption, change management, and business transformation initiatives.
What youll need to succeed (Skills & Experience)
Essential
  • 3-4 Years of experience with strong experience in AI testing, quality engineering, AI/LLM evaluation, or Agentic AI assurance.
  • Good understanding of LLMs, prompt engineering, RAG, AI agents, and multi-agent workflows.
  • Ability to define evaluation frameworks, quality metrics, KPIs, scorecards, and success criteria.
  • Hands-on experience validating AI accuracy, reasoning, reliability, hallucination risk, relevance, completeness, and business outcomes.
  • Strong analytical, problem-solving, stakeholder management, and communication skills.
Desirable
  • Minimum 3-4 years of experience with AI evaluation tools, automation frameworks, and data-driven quality assessment techniques.
  • Exposure to benchmarking models, prompts, tools, and agent configurations to improve performance.
  • Experience supporting AI adoption, user acceptance, change management, or wider business transformation initiatives.
What you will be doing (Role Accountabilities)
  • Define and govern AI quality standards, evaluation frameworks, KPIs, and success criteria.
  • Validate Agentic AI solutions for accuracy, reasoning, reliability, and business effectiveness.
  • Assess prompts, LLM outputs, and end-to-end agent workflows to ensure quality outcomes.
  • Benchmark models, prompts, tools, and agent configurations to optimize performance.
  • Measure and improve AI quality metrics, including accuracy, relevance, completeness, and hallucination rates.
  • Identify risks, quality gaps, and opportunities for continuous improvement.
  • Ensure Responsible AI, security, privacy, governance, and compliance requirements are met.
  • Support AI adoption, user acceptance, and organizational change initiatives.
  • Collaborate with Product, Engineering, Architecture, QA, and business stakeholders.
  • Present insights, risks, performance metrics, and strategic recommendations to leadership.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Test Analyst - AI Assurance
Test Analyst - AI Assurance

Hexaware Technologies Limited • India

On-site
INR 4,500,000 - 7,500,000
AI Test Engineer
AI Test Engineer

AXA Global Business Services (AXA GBS) • Pune District

On-site
INR 900,000 - 1,500,000
AI Quality Assurance Engineer
AI Quality Assurance Engineer

ThirdEye Data Inc. • Bengaluru

Hybrid
INR 1,800,000 - 3,200,000
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Mployee.me • Mumbai

On-site
INR 1,200,000 - 1,800,000
Software Development Engineer in Test – AI
Software Development Engineer in Test – AI

WebHosting • India

Remote
INR 1,500,000 - 2,300,000
AI or ML Engineer - Python, QA Testing
AI or ML Engineer - Python, QA Testing

Optum • Hyderabad

On-site
INR 1,800,000 - 3,200,000
AI SDET
AI SDET

ProductSquads Technolabs LLP • Ahmedabad District

On-site
INR 900,000 - 1,300,000
AI Evaluation Engineer
AI Evaluation Engineer

Vinyl Equity, Inc. • Mangaluru

On-site
INR 1,200,000 - 1,800,000
Competitive compensation
Equity participation
Health insurance
+1
AI Engineer
AI Engineer

Qentelli • Hyderabad

On-site
INR 4,000,000 - 7,000,000
Test Lead - AI Agent Testing
Test Lead - AI Agent Testing

Hexaware Technologies Limited • Pune District, Chennai, Bengaluru

Hybrid
INR 1,500,000 - 2,100,000