Quality Lead - AI & Automation Auditing (BPO Contact Center)

Sutherland Company; Angeles; On-site; PHP 800,000 - 1,500,000; Full time

Job summary

Sutherland Company is seeking an operational Knowledge AI Quality Lead to oversee AI-driven support tools within our BPO partner’s contact center operations in Angeles, Philippines. The role focuses on auditing, testing, and improving the performance of automated workflows, virtual agents, and human-assisted handovers to ensure accurate, compliant customer experiences.

You will work with client AI teams, KB managers, and engineering to translate audit findings into system retuning, sprint

Qualifications

• 2+ years in QA within a high-volume BPO/contact center.
• Experience auditing AI chatbots, conversational flows, or automation tools.
• Travel industry experience preferred.
• Familiarity with Lean, Six Sigma, or root-cause analysis.

Responsibilities

• Operationalize QA frameworks for AI chatbots, virtual agents, and copilot tools.
• Perform routine UAT and regression testing across deployments and KB updates.
• Audit live AI conversations and handover to Human-Assist agents for context retention.
• Synthesize audit data into weekly reports and drive improvement with Client AI Automation Leads and Product Engineering.
• Identify KB gaps and new intents to broaden containment safely.

Skills

Conversational AI

QA Testing

Data Analysis

Process Improvement

Job description

Company Description:

We are seeking an operational Knowledge AI Quality Lead from our BPO partner to oversee, monitor, and elevate the performance of our AI-driven customer service tools and hybrid support workflows. Positioned within the partner's contact center operations, this role ensures that automated workflows, virtual assistants, agent copilots, and human-assist handovers deliver reliable, accurate, and compliant customer experiences.

The ideal candidate combines operational contact center auditing expertise with a deep understanding of AI conversational dynamics, acting as the frontline bridge between real-time vendor performance data and client-side engineering, product, and knowledge teams.

Job Description:

AI QA Framework & Testing

• Operationalize and execute client-approved QA frameworks, evaluation rubrics, and test scripts tailored for AI chatbots, virtual agents, and copilot tools.

• Perform routine pre-release User Acceptance Testing (UAT) and regression testing across prompt deployments, Knowledge Base (KB) updates, and automated workflow releases within the operational environment.

Operational Auditing & Performance Monitoring

• Conduct daily audits of live AI-driven conversations, automated resolutions, and self-service interactions to detect intent misclassifications, system hallucinations, incorrect policy applications, or journey drops.

• Audit the handover bridge between AI and Human-Assist agents to evaluate context retention, CRM data pass-through accuracy, and agent handle times post-transfer.

Closed-Loop Vendor Feedback & Defect Analysis

• Perform root-cause defect analysis on resolution failures, escalation spikes, and recurring customer friction points.

• Synthesize audit data into actionable weekly reports; feed continuous improvement insights to Client AI Automation Leads, KB Managers, and Product Engineering for system retuning and sprint updates.

• Identify out-of-scope traveler queries and KB content gaps, recommending new intents and articles to safely broaden containment coverage.

Operational Metrics & SLA Compliance

• Measure, track, and report on primary operational and AI quality KPIs:

  • Intent Recognition Accuracy

  • Resolution Precision & Answer Accuracy

  • Containment Rate & Defect Rate

  • Handover Context Retention Accuracy

  • Traveler Sentiment Score (TSS) / CSAT Impact

• Hold operational teams and frontline agents accountable for adhering to updated AI copilot workflows and knowledge standards.

Qualifications:

Experience

• Contact Center QA Background: 2+ years of experience in quality management, operational auditing, or process improvement within a high-volume BPO/contact center environment.

• AI & Automation Auditing: Demonstrated experience testing, auditing, or evaluating AI chatbots, conversational flows, copilot integrations, or workflow automation tools.

• Travel Industry (Preferred): Prior experience in travel, hospitality, or tech-enabled customer support operations is strongly preferred.

• Methodology: Familiarity with operational process improvement tools (Lean, Six Sigma, Root Cause Analysis / 5 Whys).

Core Competencies & Skills

• Conversational AI Fluency: Deep understanding of natural language processing (NLP/NLU) behavior, intent mapping structures, and failure patterns (e.g., hallucination, drift, misclassification).

• Analytical & Data-Driven: Ability to turn transcript audits, CSS feedback, and performance dashboard data into actionable insights, using AI tools to support analysis and validating outputs for accuracy and relevance.

• Cross-Functional Communication: Clear written and verbal reporting skills to present quality defects and technical feedback to both BPO leadership and client technical teams.

• Quality Standard-Setting: Meticulous attention to detail with an unyielding commitment to brand trust and effortless traveler resolution.

Additional Information:

All your information will be kept confidential according to EEO guidelines.