AI QA Analyst: LLM/ML & RAG Testing

Rail Delivery Group; Greater London; On-site; GBP 44,000 - 45,000; Full time

Job summary

Rail Delivery Group seeks a Technical QA AI Analyst to join the Quasar team. The role blends traditional QA—documenting system knowledge, traceability, and testing—with assurance of AI-enabled solutions across RDG systems.

You will validate LLM-based features, ML models and RAG pipelines, ensure data quality, and test guardrails for safe railway services. As AI adoption grows, the role ensures both technical systems and AI solutions are rigorously assured before release, liaising with Service

Qualifications

• Proven experience of manual and automation testing in waterfall and agile environments with a strong acceptance test background.
• Experience testing or assuring AI-enabled solutions - LLM features, ML models or RAG pipelines, including output evaluation, hallucination and bias detection.
• A good understanding of AI/ML fundamentals: model training and evaluation metrics (accuracy, precision, recall, F1), data quality and drift, LLM concepts (context, prompts, grounding), and testing.

Responsibilities

• Build and document system knowledge in Confluence in the Quasar-prescribed format—process diagrams, mind maps and requirements that make system components and data flows clear.
• Own requirement traceability for assigned systems and support sign-off of acceptance for Quasar.
• Develop and execute test strategies for AI-enabled functionality (LLM features, ML models, RAG pipelines) with defined acceptance criteria.
• Evaluate AI outputs for accuracy, relevance, consistency and safety—identify hallucinations and bias; manage defects.
• Validate AI/ML performance using metrics (accuracy, precision, recall, F1) and monitor data quality and drift.
• Conduct adversarial and robustness testing on AI features and validate guardrails and human-in-the-loop controls.
• Contribute to AI governance evidence—risk assessments, explainability documents, model validation, and post-deployment monitoring.

Skills

Manual testing

Automation testing

End-to-end testing

AI testing

LLM features

ML models

RAG pipelines

Selenium

Robot Framework

Python

CI/CD

GitHub

GitLab

Tools

Selenium

Robot Framework

Python

ReadyAPI

SauceLabs

LoadRunner

GitHub

GitLab

Job description

Rail Delivery Group seeks a Technical QA AI Analyst to join the Quasar team. The role blends traditional QA—documenting system knowledge, traceability, and testing—with assurance of AI-enabled solutions across RDG systems.

You will validate LLM-based features, ML models and RAG pipelines, ensure data quality, and test guardrails for safe railway services. As AI adoption grows, the role ensures both technical systems and AI solutions are rigorously assured before release, liaising with Service