Ai Quality Analyst (Conversation Intelligence & Evaluation)

Dash Bpo; Gauteng; On-site; ZAR 400 000 - 600 000; Full time

Job summary

A leading AI evaluation firm in Gauteng is seeking an AI Quality Analyst to evaluate performance of AI-powered interactions. You will combine analytical precision with QA discipline to transform conversational data into actionable insights. The role requires strong analytical skills and familiarity with conversational AI tools. Join us in shaping the next generation of customer experiences.

Benefits

Shaping next generation of AI experiences

Turning conversation data into business impact

Cross-functional collaboration

Qualifications

• Strong analytical and QA background with experience in data evaluation.
• Familiarity with conversational AI platforms or QA tools.
• Exceptional attention to detail and balance of qualitative and quantitative perspectives.

Responsibilities

• Evaluate AI-handled customer interactions for accuracy and effectiveness.
• Analyze AI interaction data to uncover trends and optimization opportunities.
• Build and maintain dashboards and performance summaries for stakeholders.

Skills

Analytical precision

QA discipline

Data storytelling

Attention to detail

Communication skills

Fluency in English (C1+)

Education

Bachelor's degree in Data Analytics, Linguistics, Computer Science or related field

Tools

Excel

Power BI

Tableau

Observe.AI

Level.AI

Tethr

Job description

AI Quality Analyst (Conversation Intelligence & Evaluation)

We are seeking a detail-oriented and insight-driven AI Quality Analyst to evaluate the performance of AI-powered customer interactions across voice and chat channels.

This role focuses on ensuring that AI agents deliver accurate, empathetic, and business-aligned responses while continuously surfacing opportunities for model and workflow improvement.

The ideal candidate will combine analytical precision, QA discipline, and data storytelling to transform large volumes of conversational data into actionable insights.

You will help bridge the gap between raw AI performance metrics and meaningful business outcomes—guiding both internal stakeholders and clients on where AI is adding value and where it needs refinement.

This is not a call review position—it's an intelligence role that requires curiosity, pattern recognition, and the ability to see beyond surface-level metrics.

Key Responsibilities

• AI Interaction Evaluation: Evaluate AI-handled customer and sales interactions for accuracy, intent recognition, tone, compliance, and effectiveness.
• Identify gaps, hallucinations, misclassifications, or missed intents in AI responses.
• Provide structured and qualitative feedback on conversational quality and system performance.
• Maintain consistent scoring rubrics and calibration across evaluation sets.
• Data Analytics & Insight Generation: Analyze AI interaction data to uncover trends, recurring issues, and optimization opportunities.
• Build and maintain dashboards, scorecards, and performance summaries for internal and client stakeholders.
• Generate structured datasets from evaluations for retraining and continuous improvement loops.
• Translate quantitative findings into clear narratives and recommendations that influence roadmap decisions.
• Quality Assurance & Tooling: Operate within existing QA and conversational intelligence tools (e.g., Observe.AI, Level.AI, Tethr, custom dashboards).
• Contribute to the evolution of internal evaluation frameworks, metrics, and annotation guidelines.
• Ensure that QA processes align with business goals, compliance standards, and customer satisfaction KPIs.
• Support release sign-offs and regression checks for new conversational flows or AI model updates.
• Collaborate with AI engineers, data scientists, and client teams to communicate findings and improvement priorities.
• Present analytical insights and performance reviews in clear, client‑friendly formats.
• Contribute to ongoing refinement of QA workflows, evaluation standards, and reporting templates.
• Stay current on conversational AI trends, evaluation methodologies, and tooling innovations.
• Share learnings across teams and contribute to a culture of measurement‑driven improvement.
• Develop domain expertise across multiple client verticals (e.g., support, sales, finance, healthcare).
• Continuous Learning & Domain Growth: Stay current on conversational AI trends, evaluation methodologies, and tooling innovations; share learnings across teams.

Requirements

• Bachelor's degree in Data Analytics, Linguistics, Computer Science, or a related field (or equivalent experience).
• Strong analytical and QA background with demonstrated experience in data evaluation, quality frameworks, or conversational review.
• Familiarity with conversational AI platforms or QA tools (e.g., Observe.AI, Level.AI, Tethr, CallMiner, or custom LLM dashboards).
• Proficiency in data analysis and visualization tools (Excel, or BI platforms such as Power BI or Tableau).
• Exceptional attention to detail and ability to balance qualitative and quantitative perspectives.
• Excellent communication skills — able to synthesize complex data into clear business insights.
• Fluency in English (C1+).
• Nice to have: Experience working with conversational AI models (OpenAI, Anthropic, Google Dialogflow, etc.).
• Exposure to LLM evaluation frameworks (e.g., prompt testing, response accuracy scoring).
• Background in contact center analytics, speech analytics, or customer experience optimization.
• Knowledge of compliance and quality standards in customer operations (PCI, HIPAA, GDPR).

Benefits

• Evaluate and shape the next generation of AI‑driven customer experiences.
• Turn conversation data into measurable business impact.
• Work across analytics, QA, and client collaboration to ensure that AI agents deliver real value.
• Be at the intersection of AI performance, customer satisfaction, and business intelligence.