Quality Analyst - AI Voice Call-QC

Paytm; Dadri; On-site; INR 900,000 - 1,300,000; Full time

Job summary

Paytm is seeking a Quality Analyst - AI Voice Call QC to own the quality of merchant-facing AI conversations. You will audit AI call recordings, review transcripts against prompts and rules, and identify gaps to improve production call quality.

The role requires strong call QC, prompt engineering, and analytical thinking, with fluency in Hindi and English. This position focuses on compliance, calibration, and continuous improvement of AI campaigns.

Qualifications

• 2–5 years of experience in Quality Assurance, Call Quality Audits, Contact Centre Quality, Conversational AI, or related domain.
• Hands-on experience auditing voice calls or chat conversations against defined quality parameters and scorecards.
• Hands-on experience writing and refining prompts for AI/LLM-based applications, preferably voice agents.
• Ability to analyze AI conversations for issues related to intent understanding, prompt adherence, hallucination, tone, compliance, and outcomes.
• Fluency in Hindi and English; proficiency in regional languages is a plus.

Responsibilities

• AI Call Audits: Listen to AI voice call recordings and review transcripts daily against prompts, flows, and rules.
• Issue Identification: Identify off-script responses, hallucinations, repetition, language/script handling gaps, and poor closures.
• Prompt Writing Refinement: Write and test production prompts; fix flow, objection handling, edge cases, and language handling.
• Root Cause Analysis: Classify issues by root cause and route to the right owner.
• Pre-launch Testing: Execute scenario-based test cases before go-live.
• Post-change Validation: Validate fixes on live calls to avoid regressions.
• Quality Framework: Define QC parameters, audit checklists, and scoring rubrics.
• Compliance Monitoring: Flag compliance gaps like mis-selling or privacy lapses.
• Disposition Accuracy: Verify post-call dispositions and outcome tagging reflect what happened.
• Quality Reporting/Feedback: Publish dashboards on quality scores and improvements; share insights with teams.
• Multilingual QC: Audit calls in Hindi, English, Hinglish, and regional languages.
• Documentation Governance: Maintain audit logs, prompt history, test cases, and known issues.

Skills

Quality Assurance

Call QC

Prompt Engineering

AI Quality

Attention to Detail

Analytical Skills

Hindi

English

Communication

Execution

Tools

Excel

Google Sheets

Job description

About Us

Paytm is Indias leading mobile payments and financial services distribution company. Pioneer of the mobile QR payments revolution in India, Paytm builds technologies that help small businesses with payments and commerce. Paytm s mission is to serve half a billion Indians and bring them to the mainstream economy with the help of technology.

About the Team

The AI Operations team is responsible for building, launching, and scaling AI-powered voice and Agentic AI solutions across Paytm s merchant ecosystem.

The team works closely with Business, Product, Technology, Data, QA, and Operations teams to drive AI-led automation across merchant engagement, retention, activation, support, and service journeys.

The team owns the complete operational lifecycle of AI campaigns - from use-case identification and conversational journey design to voice-bot creation, prompt development, testing, deployment, performance monitoring, and continuous optimization.

About the Role

We are looking for a Quality Analyst - AI Voice Call QC who will own the quality of AI voice agent conversations across merchant-facing campaigns.

The role will involve listening to AI call recordings, auditing transcripts against the prompt and business rules, identifying conversation and compliance gaps, writing and refining prompts, and validating that fixes improve call quality in production.

The candidate should have a strong combination of call quality auditing, prompt engineering, conversational AI understanding, attention to detail, and analytical thinking.

Role Expectations / Responsibilities

• AI Call Audits: Listen to AI voice call recordings and review transcripts daily to evaluate agent performance against the prompt, conversation flow, business rules, and expected outcome.
• Issue Identification: Identify issues such as incorrect or off-script responses, hallucination, repetition, long monologues, missed objections, incorrect intent identification, wrong language/script handling, interruptions, unnecessary probing, and poor call closure.
• Prompt Writing Refinement: Write, test, and refine production prompts based on audit findings - fixing conversation flow, objection handling, edge cases, language handling, tool/variable usage, and closure logic.
• Root Cause Analysis: Classify each issue by root cause - prompt gap, business logic, STT/TTS error, latency, tool/API failure, or data issue - and route it to the right owner.
• Pre-launch Testing: Test new and updated AI agents before go-live using test calls and scenario-based test cases covering happy paths, objections, edge cases, and negative scenarios.
• Post-change Validation: Validate every prompt change on live calls to confirm the fix works and has not introduced regressions elsewhere in the flow.
• Quality Framework: Define and maintain QC parameters, audit checklists, scoring rubrics, and critical/non-critical error categories for AI voice calls.
• Compliance Monitoring: Flag critical compliance gaps such as mis-selling, incorrect financial information, unauthorized promises, disclosure misses, and data privacy lapses, and ensure immediate corrective action.
• Disposition Outcome Accuracy: Verify that post-call dispositions, feedback tags, and outcome capture accurately reflect what happened on the call.
• Quality Reporting: Publish regular quality reports and dashboards covering quality scores, top issue categories, trends, fixes shipped, and their impact.
• Feedback Loop: Share call-level insights with Campaign Managers, Product, and Business teams, and convert recurring issues into prompt, flow, or platform improvements.
• Multilingual QC: Audit calls across Hindi, English, Hinglish, and regional languages for accuracy, tone, pronunciation, and language-switching behaviour.
• Documentation Governance: Maintain audit logs, prompt change history, test cases, known issues, and quality benchmarks for every campaign.

Key Success Metrics

• AI call quality
• Prompt effectiveness
• Compliance
• Audit efficiency
• AI conversation quality score
• Prompt adherence rate
• Reduction in hallucinations and off-script responses
• Critical compliance error rate
• Issue detection-to-fix turnaround time
• Post-fix regression rate
• Disposition / outcome tagging accuracy
• Audit coverage and volume
• Pre-launch defect catch rate
• Improvement in campaign successful outcome rate driven by quality fixes

Required Qualifications

• Experience: 2-5 years of experience in Quality Assurance, Call Quality Audits, Contact Centre Quality, Conversational AI, or a related domain.
• Call QC Experience: Hands-on experience auditing voice calls or chat conversations against defined quality parameters and scorecards.
• Prompt Engineering: Hands-on experience writing and refining prompts for AI/LLM-based applications, preferably voice agents.
• AI Quality: Ability to analyze AI conversations and pinpoint issues related to intent understanding, prompt adherence, hallucination, repetition, tone, compliance, and outcome acheivement.
• Attention to Detail: Strong listening skills and the ability to catch subtle errors in AI responses, language, and flow.
• Analytical Skills: Strong analytical and problem-solving skills with proficiency in Excel/Google sheets.
• Languages: Fluency in Hindi and English. Proficiency in one or more regional languages will be an advantage.
• Communication: Excellent written and verbal communication skills with strong documentation capabilities.
• Execution: Strong ownership and ability to audit multiple AI campaigns in parallel with fast turnaround.

Preferred Skills

• Experience in FinTech, Payments, Lending, Merchant Operations, or Contact Centre Operations.
• Hands-on experience with AI voice agents / OBD / IVR quality audits.
• Understanding of LLMs, speech-to-text, text-to-speech, intent classification, tool calling, and agent workflows.
• Familiarity with templating in prompts (e.g., Jinja2) and structured prompt formats.
• Experience using LLMs to automate transcript analysis and QC at scale.
• Exposure to SQL, dashboards, and CRM/ticketing systems.

We may use artificial intelligence (AI) tools to support parts of the hiring process, such as reviewing applications, analyzing resumes, or assessing responses and identifying potential inconsistencies or verification signals in application materials based on available information. These tools assist our recruitment team but do not replace human judgment. Final hiring decisions are ultimately made by humans. If you would like more information about how your data is processed, please contact us.