QA Tech Analyst 12m FTC

Rail Delivery Group; Greater London; On-site; GBP 44,000 - 45,000; Full time

Job summary

Rail Delivery Group seeks a Technical QA AI Analyst to join the Quasar team. The role blends traditional QA—documenting system knowledge, traceability, and testing—with assurance of AI-enabled solutions across RDG systems.

You will validate LLM-based features, ML models and RAG pipelines, ensure data quality, and test guardrails for safe railway services. As AI adoption grows, the role ensures both technical systems and AI solutions are rigorously assured before release, liaising with Service

Qualifications

• Proven experience of manual and automation testing in waterfall and agile environments with a strong acceptance test background.
• Experience testing or assuring AI-enabled solutions - LLM features, ML models or RAG pipelines, including output evaluation, hallucination and bias detection.
• A good understanding of AI/ML fundamentals: model training and evaluation metrics (accuracy, precision, recall, F1), data quality and drift, LLM concepts (context, prompts, grounding), and testing.

Responsibilities

• Build and document system knowledge in Confluence in the Quasar-prescribed format—process diagrams, mind maps and requirements that make system components and data flows clear.
• Own requirement traceability for assigned systems and support sign-off of acceptance for Quasar.
• Develop and execute test strategies for AI-enabled functionality (LLM features, ML models, RAG pipelines) with defined acceptance criteria.
• Evaluate AI outputs for accuracy, relevance, consistency and safety—identify hallucinations and bias; manage defects.
• Validate AI/ML performance using metrics (accuracy, precision, recall, F1) and monitor data quality and drift.
• Conduct adversarial and robustness testing on AI features and validate guardrails and human-in-the-loop controls.
• Contribute to AI governance evidence—risk assessments, explainability documents, model validation, and post-deployment monitoring.

Skills

Manual testing

Automation testing

End-to-end testing

AI testing

LLM features

ML models

RAG pipelines

Selenium

Robot Framework

Python

CI/CD

GitHub

GitLab

Tools

Selenium

Robot Framework

Python

ReadyAPI

SauceLabs

LoadRunner

GitHub

GitLab

Job description

Salary Scale: £44,389 - £45,000 per annum.

RDG's Quality Assurance Practice is looking for a Technical QA AI Analyst to join the Quasar team. This is a pivotal role blending traditional technical QA - building and documenting system knowledge in the format prescribed by Quasar, maintaining clear traceability between Architectural and QA components, and conducting technical and acceptance testing - with assurance of the AI-enabled solutions being adopted across RDG's systems.

The role will define how AI solutions are tested and assured: validating LLM-based features, ML models and retrieval-augmented (RAG) pipelines, evaluating AI outputs for accuracy, relevance, consistency and safety, detecting hallucinations and bias, assuring data quality, and testing the robustness and guardrails that keep AI features safe for the railway and its customers.

As the rail industry undergoes considerable change, with transformational programmes across smart ticketing and digital services and a growing adoption of AI across RDG, this role is pivotal in ensuring both technical systems and AI solutions are rigorously assured before release.

The role will liaise with Service Assurance, Architecture, CCI, data and technology teams, and technical and management teams from TOCs, suppliers and other industry stakeholders.

What can I expect to do in this job?

• Build and document system knowledge in Confluence in the Quasar-prescribed format - process diagrams, mind maps and requirements that make system components, end-to-end data flows and dependencies clear to all team members.
• Own requirement traceability for assigned systems - baseline and re-baseline traceability matrices, keep coverage accurate through ConnectALL mappings, and support sign-off of requirement acceptance for Quasar.
• Develop and execute test strategies for AI-enabled functionality (LLM features, ML models, RAG pipelines), with acceptance criteria and test oracles suited to non-deterministic systems.
• Evaluate AI outputs for accuracy, relevance, consistency and safety - identifying hallucinations, bias and fairness issues, and managing them through standard defect management.
• Validate AI/ML performance using appropriate metrics (accuracy, precision, recall, F1), assure training and evaluation data quality, and monitor for data drift.
• Conduct adversarial and robustness testing on AI features - red-teaming, prompt injection, edge cases - and validate guardrails and human-in-the-loop controls.
• Contribute to AI governance evidence - risk assessments, explainability documentation, model validation records, and post-deployment monitoring of AI-enabled services.
• Create and deliver all key QA deliverables - test plans, strategies, scope, test objective matrices, test cases, reports and defect management - across waterfall and agile projects.
• Maintain and build functional and non-functional testing using approved automation tools - Selenium, Robot Framework, Python, ReadyAPI, SauceLabs, LoadRunner - automating regression to improve time to market.
• Manage automation scripts and code bases through GitHub and GitLab, supporting testing in a CI/CD and DevOps model, and use AI tools to augment test generation, coverage and productivity.
• Ensure supplier and third-party test delivery (including AI features) is in line with RDG policy and standards, working collaboratively with vendors for release and product quality.
• Report daily testing status to the Test Manager and stakeholders, elevate delays proactively, and share knowledge across the team on technical and AI testing subjects.

Who will my key contacts be?

• Internal: RDG's Service Assurance (QA and Accreditation) and Quasar teams.
• Internal: RDG Architecture and CCI, including AI governance and data functions.
• Internal: RDG Technology Services, RDG IT, and Programme and Project Management teams.
• Internal: internal functions including Audit, Customer Service, and Finance teams.
• External: TOCs and other stakeholders.
• External: Technology suppliers, service providers, and AI/ML solution providers.
• External: Department for Transport.
• External: Transport for London.
• External: All related consultants and contracting arrangements within a project or programme of work.

What experience, skills and knowledge do I need?

• Proven experience of manual and automation testing of end-to-end solutions, estimation, reporting and defect management, in both waterfall and agile environments, with a strong acceptance test background.
• Experience testing or assuring AI-enabled solutions - LLM features, ML models or RAG pipelines - including output evaluation, hallucination and bias detection, and guardrail validation.
• A good understanding of AI/ML fundamentals: model training and evaluation metrics (accuracy, precision, recall, F1), data quality and drift, LLM concepts (context, prompts, grounding), and testing.