Senior QA & Automation Engineering Leader – Conversation AI Assurance and Test Automation

Resillion; Glasgow; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

Resillion is seeking a Senior QA & Automation Engineering Leader to own AI testing, quality assurance and automation across enterprise-scale programmes. You will combine strategic QA leadership with hands-on technical delivery in Conversational AI, LLMs and RAG.

The role works in a hybrid model with travel to Glasgow and close collaboration with AI Engineers, Data Scientists and business stakeholders to ensure AI solutions are accurate, reliable and production-ready.

Qualifications

• Extensive senior-level QA leadership experience (15–20+ years).
• Proven hands-on experience testing Conversational AI (Chat/Voice).
• Ownership of AI testing strategy, delivery and quality outcomes.
• Experience leading AI testing teams and delivery accountability.
• Strong understanding of LLMs, RAG and AI-enabled applications.
• Experience developing AI testing frameworks, evaluation methodologies and quality metrics.
• Strong automation architecture, programming and CI/CD experience.
• Excellent stakeholder management and communication skills.
• Experience in regulated enterprise environments desirable.

Responsibilities

• Own end-to-end testing strategy for Conversational AI across chat and voice.
• Lead testing of LLM-powered applications and agentic AI workflows.
• Validate conversational accuracy, context, intent and dialogue management.
• Test voice capabilities including ASR, TTS, latency and interruptions.
• Assess AI-generated responses for accuracy, relevance, and bias.
• Validate prompts, retrieval, context, integrations and guardrails.
• Establish AI testing frameworks, benchmarks and quality metrics.
• Take ownership of AI testing strategy, delivery and quality outcomes.
• Lead QA and automation teams, setting direction and priorities.
• Drive production readiness and continuous improvement.
• Collaborate with AI Engineering, Data Science and business stakeholders.
• Communicate testing progress and risks to senior leaders.
• Architect and maintain scalable automation frameworks and CI/CD pipelines.
• Increase automation coverage and regression efficiency.
• Apply AI-assisted testing approaches with validation.

Skills

QA leadership

Automation testing

AI testing

Conversational AI

LLMs

Selenium

Playwright

Cypress

UFT

Python

Java

Cucumber BDD

CI/CD

Azure DevOps

Jenkins

GitHub Actions

Tools

Selenium

Playwright

Cypress

UFT

Job description

Senior QA & Automation Engineering Leader – Conversation AI Assurance and Test Automation

Location: Central Scotland

Working Arrangement: Hybrid, with travel to Glasgow (2 Days per week)

Job Type: Permanent or Contract (Inside IR35)

** Due to background-check requirements, applicants must have a minimum of three years' UK address history. **

About Resillion

Resillion is a global provider of digital assurance, cybersecurity, and testing services. We help organisations strengthen their security posture, protect critical assets, and deliver high-quality digital solutions. Our teams operate across multiple specialist disciplines, including Digital Forensics, Incident Response, SOC Operations, Quality & Compliance, and Technical Consultancy. We are committed to excellence, continuous improvement, and delivering trusted, customer-focused services that meet the highest professional and accreditation standards.

Role Overview

We are seeking an experienced Senior QA & Automation Engineering Leader to take ownership of AI testing, quality assurance and automation across enterprise-scale technology programmes.

The successful candidate will combine strategic QA leadership with hands‑on technical expertise, leading the testing and assurance of Conversational AI, Generative AI, Large Language Models (LLMs), Retrieval‑Augmented Generation (RAG) and agentic AI solutions.

Proven experience testing Conversational AI systems (Chat and/or Voice) is essential. Candidates must demonstrate practical experience owning AI testing, leading teams, managing delivery and driving measurable improvements in conversational AI quality.

The role will involve working closely with AI Engineers, Data Scientists, Software Engineering teams and business stakeholders to ensure AI solutions are accurate, reliable, secure and fit for production.

Key Responsibilities

Conversational AI and AI Assurance

• Own and drive the end-to-end testing strategy for Conversational AI solutions across chat and voice platforms.
• Lead testing of LLM-powered applications, chatbots, voice assistants, RAG solutions and agentic AI workflows.
• Validate conversational accuracy, contextual understanding, intent recognition, response consistency, multi‑turn interactions and dialogue management.
• Test voice capabilities, including speech recognition (ASR), text‑to‑speech (TTS), response latency and interruption handling.
• Assess AI‑generated responses for accuracy, relevance, hallucinations, bias and adherence to business requirements.
• Validate prompt handling, retrieval accuracy, context management, integrations and AI guardrails.
• Establish AI testing frameworks, evaluation benchmarks, regression suites and measurable quality metrics.

AI Testing Leadership and Delivery

• Take ownership of AI testing strategy, execution, quality outcomes and delivery accountability.
• Lead and develop QA and automation engineering teams, setting technical direction and delivery priorities.
• Drive conversational AI quality, continuous improvement and production readiness.
• Work closely with AI Engineering, Data Science, Data Engineering and business stakeholders.
• Communicate testing progress, quality risks, defects and release recommendations to senior stakeholders.

Test Automation and Engineering

• Architect and maintain scalable automation frameworks using Selenium, Playwright, Cypress, UFT, Python, Java and Cucumber BDD.
• Develop automated UI, API, integration, mobile, backend and data validation test suites.
• Integrate automated testing into Azure DevOps, Jenkins and GitHub Actions CI/CD pipelines.
• Drive automation coverage, regression efficiency and continuous testing.
• Apply AI‑assisted testing approaches using GitHub Copilot and Microsoft Copilot, with appropriate validation.

Data, API and Enterprise Assurance

• Lead data quality, integrity, profiling, reconciliation and ETL validation activities.
• Validate APIs, integrations, data migrations and end‑to‑end enterprise workflows.
• Support performance, resilience and security testing using tools such as JMeter and established OWASP principles.
• Establish risk‑based testing strategies, governance standards, requirements traceability and evidence‑based release acceptance criteria.

Technical Skills

Conversational AI and AI Assurance: Chat and Voice AI testing, LLM evaluation, RAG, agentic AI, hallucination detection, prompt testing, ASR, TTS, AI guardrails and model validation.

Test Automation: Selenium WebDriver, Playwright, Cypress, UFT, Appium, Cucumber BDD, PyTest.

API and Integration: Postman, RestAssured, SoapUI, REST, SOAP, OAuth 2.0.

CI/CD and DevOps: Azure DevOps, Jenkins, GitHub Actions, Git, Docker, Maven and Gradle.

Data Platforms: SQL, Oracle, DB2, MySQL, Databricks and ETL validation.

Performance and Security: JMeter, LoadRunner and OWASP principles.

Governance: Risk‑based testing, Agile, Scrum, SAFe, Jira, Xray, Confluence and ServiceNow.

Essential Experience and Requirements

• Extensive senior‑level QA, test automation or quality engineering leadership experience, ideally 15–20+ years.
• Proven hands‑on experience testing Conversational AI systems (Chat and/or Voice).
• Demonstrable ownership of AI testing strategy, delivery and quality outcomes.
• Experience leading AI testing teams and managing delivery accountability.
• Strong understanding of LLMs, RAG, conversational AI and AI‑enabled applications.
• Experience developing AI testing frameworks, evaluation methodologies and quality metrics.
• Strong automation architecture, programming and CI/CD experience.
• Excellent stakeholder management, communication and leadership capabilities.
• Experience within financial services, pensions, banking, insurance or other regulated enterprise environments is desirable