Senior QAT (Quality Assurance Tester)

Scrumconnect Limited; Newcastle upon Tyne; On-site; GBP 60,000 - 85,000; Full time

Job summary

Scrumconnect Consulting in Newcastle, United Kingdom, is seeking an experienced Senior Quality Assurance Tester to lead testing strategy for a government AI platform. Responsibilities include evaluating AI output quality, API testing, and CI/CD integration within agile teams.

You will design LLM evaluations, oversee non-functional testing, and ensure governance compliance across the delivery lifecycle, working in secure government environments.

Qualifications

• Experience in a modern programming language (preferably Java or Python).
• Experience in manual API testing and tools.
• Experience building and maintaining API automation testing, preferably with REST Assured or similar tools.
• Adaptable team player with shift-left principles and CI/CD tools like GitLab/Jenkins.
• Experience conducting non-functional testing (performance and accessibility).
• Experience working in an agile environment with multiple stakeholders.
• Comfortable in secure, regulated environments handling sensitive data.
• Experience evaluating AI output quality (hallucination, relevance, retrieval).
• Experience delivering digital services in UK central government or public sector.
• Familiarity with GOV.UK Service Standard expectations and governance.
• Exposure to open-source LLM tooling or evaluation frameworks.
• Familiarity with AI/ML tooling on AWS, Azure or GCP.

Responsibilities

• Define and own the test strategy across functional, non-functional and AI-specific quality dimensions (answer quality, retrieval accuracy, hallucination rate).
• Design and run structured evaluations of LLM outputs with engineering to track quality and flag regressions.
• Carry out manual and automated API testing across application logic connecting RAG pipelines to systems.
• Build and maintain automated test suites integrated into CI/CD pipelines.
• Perform performance testing and accessibility testing to ensure robustness and inclusivity.
• Verify guardrails and human-in-the-loop checkpoints work as intended.
• Collaborate with Product, Engineering and Policy stakeholders to embed testing in sprints.
• Ensure testing and release evidence aligns with GOV.UK standards and governance.
• Contribute test plans and evaluation results to Docs-as-Code artefacts for knowledge transfer.

Skills

Programming (Java/Python)

Manual API testing

API automation testing (REST Assured)

Shift-left / CI/CD

CI/CD tools (GitLab, Jenkins)

Non-functional testing (performance, 1

Agile delivery teamwork

AI/Large Language Model evaluation

Security/regulatory environment

Open-source LLM tooling exposure

Cloud familiarity (AWS/Azure/GCP)

Tools

GitLab

Jenkins

REST Assured

k6

Gatling

JMeter

Job description

Newcastle, United Kingdom | Posted on 16/10/2026

Scrumconnect Consulting is a multi-award-winning UK public sector digital consultancy, positively influencing the lives of over forty million UK citizens. We bring deep expertise in agile delivery, digital transformation, and platform engineering across central government. With a team of over one hundred and twenty-five professionals — more than forty percent of whom are women — we are committed to building inclusive, diverse teams that reflect the communities we serve.

Overview

We are seeking an experienced Senior Quality Assurance Tester to lead testing strategy and execution for a Generative AI platform for a UK government department. This role goes beyond traditional functional QA. It includes evaluating AI output quality (answer accuracy, retrieval relevance, hallucination rate) alongside conventional API, integration and non-functional testing.

Key Responsibilities

Define and own the test strategy across functional, non-functional and AI-specific quality dimensions (answer quality, retrieval accuracy, hallucination rate), covering RAG pipelines, prompt templates and Agentic workflows.

AI Output Evaluation

Design and run structured evaluations of LLM outputs, working with engineering to track answer quality and hallucination rate over time, and flag regressions before release.

API & Integration Testing

Carry out manual and automated API testing (for example with REST Assured or similar tools) across the application logic that connects RAG pipelines to legacy and multi-cloud systems.

Automation & CI/CD Integration

Build and maintain automated test suites integrated into CI/CD pipelines (for example GitLab or Jenkins), enabling shift-left testing across the delivery lifecycle.

Non-Functional Testing

Carry out performance testing (for example k6, Gatling or JMeter) and accessibility testing, so the platform is robust, responsive and inclusive under real government usage conditions.

Responsible AI Testing

Test that guardrails and human-in-the-loop checkpoints defined by policy and technical leads work correctly and consistently. This is a verification role, not a policy-setting one.

Work within agile delivery teams

Engaging with Product, Engineering and Policy stakeholders to embed testing throughout the sprint rather than treating it as a final gate.

Governance & Standards Compliance

Make sure testing and release evidence aligns with the GOV.UK Service Standard and departmental governance, supporting evidence-gathering for service assessments where required.

Documentation & Handover

Contribute test plans, evaluation frameworks and results to "Docs-as-Code" artefacts supporting the OKUA knowledge-transfer framework, so internal government engineers can understand how quality was assured.

Required Skills & Experience

• Experience in a modern programming language (preferably Java or Python), version control and coding standards
• Experience in manual API testing and tools
• Experience building and maintaining API automation testing, preferably with REST Assured or similar tools
• Adaptable team player, experienced in shift-left principles, including CI/CD tools such as GitLab, Jenkins or similar
• Experience conducting non-functional testing, including performance (k6, Gatling, JMeter or similar) and/or accessibility testing
• Experience working in an agile environment, with demonstrated testing activities and engagement with multiple stakeholders
• Comfortable working in secure, regulated environments and handling sensitive data appropriately
• Experience evaluating LLM or AI output quality (hallucination detection, answer relevance scoring, retrieval accuracy)
• Experience delivering digital services within UK central government or the wider public sector
• Familiarity with GOV.UK Service Standard expectations and service assessment processes
• Exposure to open-source LLM tooling or evaluation frameworks
• Familiarity with AI/ML tooling on AWS, Azure or GCP

Diversity and Inclusion

At Scrumconnect Consulting, we believe that diversity drives innovation. We are committed to creating an inclusive environment where every individual is respected, valued, and supported. We welcome applications from candidates of all backgrounds and experiences, and we actively encourage applications from women, people with disabilities, underrepresented communities, and those seeking flexible working arrangements.