AI QA Analyst for AI/ML Testing & QA Strategy

Rail Delivery Group; England; On-site; GBP 44,000 - 45,000; Full time

Benefits offered by this job

Rail travel discount

30 days annual leave

Personal development days

Season ticket loan

Pension scheme

Private medical insurance

Flexible benefits portal

Flexible working arrangements

Job summary

Rail Delivery Group is hiring a Technical QA AI Analyst to join the Quasar team. The role blends traditional QA with testing of AI-enabled solutions, including LLM features, ML models and RAG pipelines, ensuring accuracy, safety and data quality across RDG systems.

You will define AI testing approaches, manage traceability, and collaborate with multiple internal teams and external vendors to assure high-quality releases in a changing rail landscape.

Qualifications

• Proven experience of manual and automation testing in waterfall and agile environments.
• Experience testing AI-enabled solutions – LLM features and RAG pipelines.
• A solid understanding of AI/ML fundamentals, metrics and data quality.
• Familiarity with CI/CD and code management through GitHub/GitLab.
• Ability to draft clear, testable requirements and acceptance criteria.
• Strong communication skills to explain AI behaviour to non-technical audiences.

Responsibilities

• Build and document system knowledge in Confluence for Quasar-sized formats.
• Own requirement traceability, baselining, and sign-off of acceptance for Quasar.
• Develop and execute test strategies for AI-enabled functionality.
• Evaluate AI outputs for accuracy, bias, safety and reliability.
• Validate AI/ML performance using metrics and monitor data drift.
• Conduct adversarial and robustness testing and validate guardrails.
• Contribute to AI governance evidence and post-deployment monitoring.
• Create and deliver QA deliverables across waterfall and agile projects.
• Maintain functional/non-functional testing with Selenium, Robot Framework, Python, ReadyAPI.
• Manage automation scripts in GitHub/GitLab within CI/CD and DevOps.
• Coordinate supplier/third-party test delivery for AI features.
• Report daily testing status to the Test Manager and stakeholders.

Skills

Manual testing

Automation testing

AI testing

LLM testing

RAG pipelines

CI/CD

GitHub/GitLab

Selenium

Robot Framework

Python

Confluence/JIRA

Test strategy

Documentation

Education

Bachelor’s degree in Information Technology or related Engineering

ISTQB/ISEB certification

ISTQB Certified Tester – AI Testing (CT‑AI)

Tools

Selenium

Robot Framework

Python

ReadyAPI

SauceLabs

LoadRunner

GitHub

GitLab

Confluence

JIRA

ALM

PyCharm

Job description

Rail Delivery Group is hiring a Technical QA AI Analyst to join the Quasar team. The role blends traditional QA with testing of AI-enabled solutions, including LLM features, ML models and RAG pipelines, ensuring accuracy, safety and data quality across RDG systems.

You will define AI testing approaches, manage traceability, and collaborate with multiple internal teams and external vendors to assure high-quality releases in a changing rail landscape.