Quality Analyst-AI Voice Call-QC

Lever, Inc.; Dadri; On-site; INR 700,000 - 1,200,000; Full time

Job summary

Paytm is seeking a Quality Analyst – AI Voice Call QC to own the quality of AI voice agent conversations across merchant campaigns. You will audit AI call recordings, review transcripts against prompts and business rules, and identify conversation gaps for improvement.

The role requires prompt writing, root-cause analysis, and validation of fixes in production with strong attention to detail, analytical thinking, and fluency in Hindi and English.

Qualifications

• 2–5 years of experience in Quality Assurance, Call Quality Audits, or related domain.
• Hands-on experience auditing voice calls or chat transcripts against defined quality parameters.
• Hands-on experience writing and refining prompts for AI/LLM-based applications, preferably voice agents.
• Ability to analyze AI conversations for intent understanding, prompt adherence, and compliance.
• Strong listening skills, attention to detail, and proficiency in Excel/Google Sheets.

Responsibilities

• AI Call Audits: Listen to AI voice call recordings and review transcripts daily to assess quality against prompts and rules.
• Issue Identification: Detect mis-scripts, hallucinations, repetitive responses, and improper language handling.
• Prompt Writing & Refinement: Write and test production prompts based on audit findings and edge cases.
• Root Cause Analysis: Classify issues by root cause and route to owners.
• Pre-launch Testing: Test new agents with scenario-based tests before go-live.
• Post-change Validation: Verify prompt changes on live calls to prevent regressions.
• Quality Framework: Define QC parameters, checklists, and error categories.
• Compliance Monitoring: Flag critical gaps like mis-selling or data privacy lapses.
• Disposition & Outcome Accuracy: Ensure post-call tagging reflects actual outcomes.
• Quality Reporting: Publish dashboards on scores, trends, and fixes.
• Multilingual QC: Audit calls across Hindi, English, Hinglish, and regional languages.
• Documentation & Governance: Maintain audit logs and change history.

Skills

Quality Assurance

Call QC

Prompt Engineering

Analytical Skills

Excel/Sheets

Communication

Attention to Detail

Execution

Multilingual

Tools

Google Sheets

Excel

SQL

Dashboards

Job description

About Us:

Paytm is India's leading mobile payments and financial services distribution company. Pioneer of the mobile QR payments revolution in India, Paytm builds technologies that help small businesses with payments and commerce. Paytm’s mission is to serve half a billion Indians and bring them to the mainstream economy with the help of technology.

About the Team:

The AI Operations team is responsible for building, launching, and scaling AI-powered voice and Agentic AI solutions across Paytm’s merchant ecosystem.

The team works closely with Business, Product, Technology, Data, QA, and Operations teams to drive AI-led automation across merchant engagement, retention, activation, support, and service journeys.

The team owns the complete operational lifecycle of AI campaigns — from use‑case identification and conversational journey design to voice‑bot creation, prompt development, testing, deployment, performance monitoring, and continuous optimization.

About the Role:

We are looking for a Quality Analyst – AI Voice Call QC who will own the quality of AI voice agent conversations across merchant-facing campaigns.

The role will involve listening to AI call recordings, auditing transcripts against the prompt and business rules, identifying conversation and compliance gaps, writing and refining prompts, and validating that fixes improve call quality in production.

The candidate should have a strong combination of call quality auditing, prompt engineering, conversational AI understanding, attention to detail, and analytical thinking.

Role Expectations / Responsibilities:

• AI Call Audits: Listen to AI voice call recordings and review transcripts daily to evaluate agent performance against the prompt, conversation flow, business rules, and expected outcome.
• Issue Identification: Identify issues such as incorrect or off‑script responses, hallucination, repetition, long monologues, missed objections, incorrect intent identification, wrong language/script handling, interruptions, unnecessary probing, and poor call closure.
• Prompt Writing & Refinement: Write, test, and refine production prompts based on audit findings — fixing conversation flow, objection handling, edge cases, language handling, tool/variable usage, and closure logic.
• Root Cause Analysis: Classify each issue by root cause — prompt gap, business logic, STT/TTS error, latency, tool/API failure, or data issue — and route it to the right owner.
• Pre-launch Testing: Test new and updated AI agents before go-live using test calls and scenario‑based test cases covering happy paths, objections, edge cases, and negative scenarios.
• Post-change Validation: Validate every prompt change on live calls to confirm the fix works and has not introduced regressions elsewhere in the flow.
• Quality Framework: Define and maintain QC parameters, audit checklists, scoring rubrics, and critical/non‑critical error categories for AI voice calls.
• Compliance Monitoring: Flag critical compliance gaps such as mis‑selling, incorrect financial information, unauthorized promises, disclosure misses, and data privacy lapses, and ensure immediate corrective action.
• Disposition & Outcome Accuracy: Verify that post‑call dispositions, feedback tags, and outcome capture accurately reflect what happened on the call.
• Quality Reporting: Publish regular quality reports and dashboards covering quality scores, top issue categories, trends, fixes shipped, and their impact.
• Feedback Loop: Share call‑level insights with Campaign Managers, Product, and Business teams, and convert recurring issues into prompt, flow, or platform improvements.
• Multilingual QC: Audit calls across Hindi, English, Hinglish, and regional languages for accuracy, tone, pronunciation, and language‑switching behaviour.
• Documentation & Governance: Maintain audit logs, prompt change history, test cases, known issues, and quality benchmarks for every campaign.

Key Success Metrics:

• The role will be measured across AI call quality, prompt effectiveness, compliance, and audit efficiency, including:
• AI conversation quality score
• Prompt adherence rate
• Reduction in hallucinations and off‑script responses
• Critical compliance error rateIssue detection‑to‑fix turnaround time
• Post‑fix regression rate
• Disposition / outcome tagging accuracy
• Audit coverage and volume
• Pre‑launch defect catch rate
• Improvement in campaign successful outcome rate driven by quality fixes

Required Qualifications:

• Experience: 2–5 years of experience in Quality Assurance, Call Quality Audits, Contact Centre Quality, Conversational AI, or a related domain.
• Call QC Experience: Hands‑on experience auditing voice calls or chat conversations against defined quality parameters and scorecards.
• Prompt Engineering: Hands‑on experience writing and refining prompts for AI/LLM‑based applications, preferably voice agents.
• AI Quality: Ability to analyze AI conversations and pinpoint issues related to intent understanding, prompt adherence, hallucination, repetition, tone, compliance, and outcome acheivement.
• Attention to Detail: Strong listening skills and the ability to catch subtle errors in AI responses, language, and flow.
• Analytical Skills: Strong analytical and problem‑solving skills with proficiency in Excel/Google sheets.
• Languages: Fluency in Hindi and English. Proficiency in one or more regional languages will be an advantage.
• Communication: Excellent written and verbal communication skills with strong documentation capabilities.
• Execution: Strong ownership and ability to audit multiple AI campaigns in parallel with fast turnaround.

Preferred Skills:

• Experience in FinTech, Payments, Lending, Merchant Operations, or Contact Centre Operations.
• Hands‑on experience with AI voice agents / OBD / IVR quality audits.
• Understanding of LLMs, speech‑to‑text, text‑to‑speech, intent classification, tool calling, and agent workflows.
• Familiarity with templating in prompts (e.g., Jinja2) and structured prompt formats.
• Experience using LLMs to automate transcript analysis and QC at scale.
• Exposure to SQL, dashboards, and CRM/ticketing systems.