Service Desk Artificial Intelligence Quality Analyst

Willis Towers Watson; Thane; On-site; INR 700,000 - 1,100,000; Full time

Job summary

Willis Towers Watson in Thane, India seeks an AI Quality & Performance Management Specialist for the Service Desk. You will monitor AI performance, analyze incidents, and drive continual improvement across knowledge, automation, and operations.

You will partner with Knowledge Management, Automation, and Service Desk teams to translate data into actionable insights and measurable service outcomes. A strong background in ITSM and data analysis is required.

Qualifications

• 4-6 years of experience in IT Service Desk, IT operations, or ITSM environments.
• Practical understanding of incident and service requests management.
• Experience with ServiceNow or comparable ITSM platforms preferred.
• Experience analyzing operational data, ticket trends, or service performance is advantageous.

Responsibilities

• Monitor and report AI performance indicators (deflection, resolution, escalation, fallback, containment).
• Review AI interactions to identify quality issues, failed intents, and incomplete responses.
• Analyze AI failure scenarios to identify root causes and recurring patterns.
• Identify improvements to AI intents, conversational flows, and knowledge sources.
• Validate AI improvements and monitor performance after changes.
• Maintain records of AI quality issues and improvement opportunities.
• Create and maintain dashboards and analytical views using Excel/Power BI.
• Translate data into clear insights and actionable improvements.
• Support testing and validation of new AI/automation capabilities.

Skills

ITSM knowledge

Data analysis

Quality assurance

Cross-functional collaboration

Adaptability

Tools

Excel

Power BI

ServiceNow

Job description

Description

About WTW
WTW is a leading global advisory, broking and solutions company that helps clients around the world turn risk into a path for growth. With roots dating to 1828, WTW has 49,000 employees serving more than 140 countries. We design and deliver solutions that manage risk, optimize benefits, cultivate talent, and expand the power of capital to protect and strengthen institutions and individuals. Our unique perspective allows us to see the critical intersections between talent, assets and ideas - the dynamic formula that drives business performance. Together, we unlock potential. Learn more at https://www.wtwco.com/en-us.

About the Team
The Enterprise Technology team provides core Information Technology (IT) operations and end-user technology services that support colleagues across the organization. The team focuses on keeping enterprise technology services available, secure, and well-governed, ensuring smooth day-to-day IT operations.

Summary
The Service Desk Artificial Intelligence Quality Analyst is responsible for ensuring that AI-assisted Service Desk capabilities deliver effective, reliable, and measurable support outcomes. The role combines traditional IT Service Desk operations with AI quality monitoring, data analysis, user experience assessment, and continual service improvement.

Role
AI Quality & Performance Management

• Monitor and report on key Artificial Intelligence (AI) performance indicators, including deflection, resolution, escalation, fallback, and containment rates.
• Review AI interactions and identify quality issues, failed intents, and inappropriate or incomplete responses.
• Categorize and analyze AI failure scenarios to identify root causes and recurring patterns.
• Identify opportunities to improve AI intents, conversational flows, knowledge sources, and automation.
• Validate the effectiveness of AI improvements and monitor performance following changes.
• Maintain accurate records of AI quality issues, trends, and improvement opportunities.
• Data Analysis & Reporting
• Analyze Service Desk ticket data, AI interaction data, and user feedback to identify trends and service gaps.
• Develop regular AI quality and Service Desk performance reports.
• Create and maintain dashboards and analytical views using Excel, Power BI, or equivalent tools.
• Translate data into clear insights, recommendations, and actionable improvement initiatives.
• Track key performance indicators and identify areas requiring further investigation.
• Knowledge & Automation Improvement
• Identify knowledge gaps that contribute to AI failures or poor user outcomes.
• Recommend improvements to knowledge articles, categorization, intents, and automation coverage.
• Provide structured feedback to Knowledge Management, Automation, AI, and Service Desk teams.
• Support testing and validation of new or enhanced AI and automation capabilities.
• Help ensure knowledge and automation solutions reflect actual user needs and Service Desk demand patterns.
• Service Desk Operations
• Handle incidents, service requests, and user support inquiries as required.
• Participate in Service Desk queue management and operational activities.
• Apply standard ITSM processes when handling incidents and requests.
• Escalate issues appropriately and maintain accurate ticket information.
• Use operational experience to identify recurring issues and opportunities for AI or process improvement.
• Continual Service Improvement
• Identify recurring Service Desk and AI-related issues and recommend improvement actions.
• Participate in service improvement initiatives and working groups.
• Monitor the outcomes of improvement initiatives and provide evidence of their effectiveness.
• Promote a culture of data-driven, user-focused continual improvement.
• Share insights and trends with relevant stakeholders to support informed decision-making.

Requirements
Education & Experience

Education & Experience

• At least 4-6 years' experience working in an IT Service Desk, IT operations, or IT Service Management (ITSM) environment.
• Practical understanding of incident and service request management.
• Experience with ServiceNow or a comparable ITSM platform is preferred.
• Experience analyzing operational data, ticket trends, or service performance is preferred.
• Experience working with AI-assisted support, chatbots, automation, or conversational AI is advantageous.