AI Test Lead — Hybrid, Responsible AI & QA

Yorkshire Building Society; Leeds; Hybrid; GBP 65,000 - 90,000; Full time

Benefits offered by this job

Holiday entitlement

Annual bonus

Pension plan

Private medical insurance

Benefits portal

Job summary

Yorkshire Building Society is seeking an AI Test Lead to define and lead testing and quality assurance for AI-enabled systems. This hybrid role involves working from home with onsite presence in Bradford and Leeds offices.

The role focuses on testing AI with probabilistic behaviour, bias and risk testing, and integration into DevOps/MLOps pipelines within a regulated financial services environment.

Qualifications

• Strong experience in testing and quality assurance for complex systems.
• Working knowledge of AI and ML technologies, including large language models and Retrieval-Augmented Generation (RAG).
• Familiarity with test automation, DevOps pipelines, and MLOps practices.
• Experience working with Microsoft Azure, Azure AI services, and Copilot-based solutions.
• Strong collaboration and communication skills, including engagement with non-technical stakeholders.

Responsibilities

• Develop AI testing frameworks and methodologies that account for non-deterministic behaviour, probabilistic outputs, and variability in AI responses.
• Embed responsible AI and risk testing by defining test cases that assess bias, fairness, transparency, safety, and the effectiveness of AI guardrails.
• Implement automated AI testing capabilities by developing test automation and integrating testing activities into DevOps and MLOps pipelines.
• Collaborate across delivery and governance teams to ensure testing practices align with engineering standards, regulatory expectations, and enterprise assurance requirements.

Skills

QA for complex systems

AI/ML knowledge

Test automation

DevOps practices

MLOps

Collaboration with stakeholders

Tools

Azure

Copilot-based solutions

Job description

Yorkshire Building Society is seeking an AI Test Lead to define and lead testing and quality assurance for AI-enabled systems. This hybrid role involves working from home with onsite presence in Bradford and Leeds offices.

The role focuses on testing AI with probabilistic behaviour, bias and risk testing, and integration into DevOps/MLOps pipelines within a regulated financial services environment.