AI Conversation Analyst

Revrag Ai; Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

RevRag AI is seeking an AI Conversation Analyst to guard the quality of our AI agents. You will deeply analyze conversations, identify why interactions succeed or fail, and work with product and engineering to improve agent performance.

You will review hundreds of conversations to surface patterns, build evaluation frameworks, and deliver actionable insights that reduce failures and improve quality scores across voice and chat interactions. This role thrives in a fast-paced startup environment.

Qualifications

• Strong analytical and problem-solving skills with excellent attention to detail.
• Strong written and verbal communication, and the ability to work with both technical and business teams.
• The ability to spot subtle conversation issues and explain why they matter.
• Structured thinking and strong documentation skills.
• Genuine curiosity about AI and Large Language Models.

Responsibilities

• Analyse AI conversations: review AI-generated voice calls and in-app conversations and evaluate quality across accuracy, context understanding, relevance, naturalness, tone and empathy, conversation flow, goal completion, instruction adherence, objection handling, and customer satisfaction.
• Investigate AI failures: identify and document hallucinations, gibberish responses, incorrect information, context loss, repetitive replies, poor reasoning, incorrect pronunciation, speech interruptions, misread intent, weak objection handling, conversation loops, and failures to complete business objectives.
• Perform root cause analysis: pinpoint the underlying cause, whether prompt-related, knowledge base gaps, tool failures, context management problems, speech recognition errors, text-to-speech issues, workflow limitations, or product bugs.
• Identify patterns at scale: review hundreds or thousands of conversations to surface recurring failure modes, customer-specific issues, quality trends, and emerging edge cases.
• Build quality frameworks: contribute to conversation scorecards, evaluation rubrics, quality benchmarks, failure taxonomies, review guidelines, and human-in-the-loop evaluation processes.
• Deliver actionable insights: prepare recurring quality reports covering failure categories, severity, trends, agent performance, and improvement recommendations.
• Work with AI to improve AI: use LLMs, transcription tools, analytics platforms, and internal observability tools to make analysis faster and more scalable.

Skills

Analytical thinking

Problem solving

Attention to detail

Written communication

Verbal communication

Independent ownership

Structured thinking

Curiosity about AI

Tools

Excel

Google Sheets

SQL

Python

Job description

Job Summary

We are looking for an AI Conversation Analyst to become the quality guardian of our AI agents: someone who can deeply analyse conversations, identify why an interaction succeeded or failed, uncover recurring patterns, and work with product and engineering teams to continuously improve agent performance.

If you are naturally curious about how AI thinks, enjoy analysing conversations, and love solving complex problems, this is the role for you. If you have ever wondered how we know whether an AI actually had a good conversation, this is where you answer that question every day, and help us make every AI conversation better than the last.

Responsibilities

• Analyse AI conversations: review AI-generated voice calls and in-app conversations and evaluate quality across accuracy, context understanding, relevance, naturalness, tone and empathy, conversation flow, goal completion, instruction adherence, objection handling, and customer satisfaction.
• Investigate AI failures: identify and document hallucinations, gibberish responses, incorrect information, context loss, repetitive replies, poor reasoning, incorrect pronunciation, speech interruptions, misread intent, weak objection handling, conversation loops, and failures to complete business objectives.
• Perform root cause analysis: pinpoint the underlying cause, whether prompt-related, knowledge base gaps, tool failures, context management problems, speech recognition errors, text-to-speech issues, workflow limitations, or product bugs.
• Identify patterns at scale: review hundreds or thousands of conversations to surface recurring failure modes, customer-specific issues, quality trends, and emerging edge cases.
• Build quality frameworks: contribute to conversation scorecards, evaluation rubrics, quality benchmarks, failure taxonomies, review guidelines, and human-in-the-loop evaluation processes.
• Deliver actionable insights: prepare recurring quality reports covering failure categories, severity, trends, agent performance, and improvement recommendations.
• Work with AI to improve AI: use LLMs, transcription tools, analytics platforms, and internal observability tools to make analysis faster and more scalable.

What Were Looking For

• Youre a great fit if you have:
• Strong analytical and problem-solving skills, with excellent attention to detail.
• Strong written and verbal communication, and the ability to work with both technical and business teams.
• The ability to spot subtle conversation issues and explain why they matter.
• Structured thinking and strong documentation skills.
• Genuine curiosity about AI and Large Language Models.
• The ability to work independently with high ownership, and to review large volumes of conversations.
• Comfort thriving in a fast-paced startup environment.

Nice to Have

• Experience with Voice QA or Customer Support QA.
• Experience with AI products, chatbots, or conversational AI.
• Familiarity with ChatGPT, Claude, Gemini, or similar LLMs, and concepts like prompting, hallucinations, context windows, and AI evaluation.
• Experience with Excel, Google Sheets, SQL, or Python.
• Experience designing QA scorecards or evaluation frameworks.
• Experience working with enterprise customers.

Success in This Role

Your impact will be measured by:

• The quality and accuracy of your conversation reviews and root-cause identification.
• The number of actionable insights you deliver, and the improvements implemented from them.
• A measurable reduction in recurring AI failures and improvement in AI quality scores.
• Healthy review turnaround time and conversation coverage.
• Client satisfaction with your quality reports.

Why Join RevRag AI

• Work on cutting-edge Voice and Conversational AI used in production.
• Learn directly from experienced AI engineers and product leaders.
• Influence how enterprise AI agents are built, evaluated, and improved.
• Gain exposure to LLMs, AI evaluation, prompt engineering, and observability.
• Be part of a fast-growing startup solving real customer problems with AI.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.