Head of Conversational AI QA & Automation

Resillion; Glasgow; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

Resillion is seeking a Senior QA & Automation Engineering Leader to own AI testing, quality assurance and automation across enterprise-scale programmes. You will combine strategic QA leadership with hands-on technical delivery in Conversational AI, LLMs and RAG.

The role works in a hybrid model with travel to Glasgow and close collaboration with AI Engineers, Data Scientists and business stakeholders to ensure AI solutions are accurate, reliable and production-ready.

Qualifications

• Extensive senior-level QA leadership experience (15–20+ years).
• Proven hands-on experience testing Conversational AI (Chat/Voice).
• Ownership of AI testing strategy, delivery and quality outcomes.
• Experience leading AI testing teams and delivery accountability.
• Strong understanding of LLMs, RAG and AI-enabled applications.
• Experience developing AI testing frameworks, evaluation methodologies and quality metrics.
• Strong automation architecture, programming and CI/CD experience.
• Excellent stakeholder management and communication skills.
• Experience in regulated enterprise environments desirable.

Responsibilities

• Own end-to-end testing strategy for Conversational AI across chat and voice.
• Lead testing of LLM-powered applications and agentic AI workflows.
• Validate conversational accuracy, context, intent and dialogue management.
• Test voice capabilities including ASR, TTS, latency and interruptions.
• Assess AI-generated responses for accuracy, relevance, and bias.
• Validate prompts, retrieval, context, integrations and guardrails.
• Establish AI testing frameworks, benchmarks and quality metrics.
• Take ownership of AI testing strategy, delivery and quality outcomes.
• Lead QA and automation teams, setting direction and priorities.
• Drive production readiness and continuous improvement.
• Collaborate with AI Engineering, Data Science and business stakeholders.
• Communicate testing progress and risks to senior leaders.
• Architect and maintain scalable automation frameworks and CI/CD pipelines.
• Increase automation coverage and regression efficiency.
• Apply AI-assisted testing approaches with validation.

Skills

QA leadership

Automation testing

AI testing

Conversational AI

LLMs

Selenium

Playwright

Cypress

UFT

Python

Java

Cucumber BDD

CI/CD

Azure DevOps

Jenkins

GitHub Actions

Tools

Selenium

Playwright

Cypress

UFT

Job description

Resillion is seeking a Senior QA & Automation Engineering Leader to own AI testing, quality assurance and automation across enterprise-scale programmes. You will combine strategic QA leadership with hands-on technical delivery in Conversational AI, LLMs and RAG.

The role works in a hybrid model with travel to Glasgow and close collaboration with AI Engineers, Data Scientists and business stakeholders to ensure AI solutions are accurate, reliable and production-ready.