Data Test Engineer (SC Cleared)

Scrumconnect

United Kingdom

Remote

GBP 90,000 - 140,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Scrumconnect Consulting, a UK-based digital transformation consultancy, is seeking an experienced Test Manager to lead QA and testing within a large-scale cloud data programme. The role is fully remote, based in India, supporting UK operations and engagements.

You will define the test strategy across CI/CD pipelines, data workflows, and AWS services; mentor QA teams; collaborate with DevOps and engineers; and ensure quality is embedded throughout the software delivery lifecycle.

Qualifications

  • Proven experience as Test Manager or QA Lead in cloud and data engineering environments.
  • Experience automated testing for cloud-native and data-centric platforms.
  • Experience testing large-scale data pipelines, transformation processes, and distributed systems.
  • Experience leading QA/testing teams in Agile delivery environments.
  • Strong stakeholder management and communication skills.
  • SFIA Level 5 leadership and driving quality improvements.
  • Active SC clearance (mandatory).

Responsibilities

  • Define, implement, and manage the overall test strategy across cloud-native data platforms and applications.
  • Lead end-to-end testing activities across data pipelines, cloud services, APIs, and distributed systems.
  • Establish quality standards, governance processes, and testing best practices across teams.
  • Drive a quality-first engineering culture across the programme.
  • Provide leadership, mentoring, and coaching to Test Engineers and QA team members.
  • Collaborate with DevOps and engineering to improve release quality and deployment confidence.
  • Ensure automated test execution within GitLab CI/CD pipelines.
  • Oversee data validation and testing on AWS EMR and Airflow workflows.

Skills

Python
SQL
Java
Bash
Test Automation
Data testing
CI/CD coordination
Stakeholder communication
Mentoring
Root cause analysis

Tools

Terraform
Docker
GitLab CI/CD
Jupyter Notebooks
Apache Airflow
PySpark
Spark / PySpark
Terraform
AWS EMR

Job description

Newcastle upon Tyne, United Kingdom | Posted on 17/07/2026

Scrumconnect Consulting is a UK-based digital transformation consultancy delivering agile, secure, and scalable technology solutions for public and private sector clients. This is a fully remote role based in India, supporting UK operations and client engagements. The successful candidate will work closely with UK leadership teams and must be comfortable operating within UK regulatory, governance, and delivery frameworks.

About the Role

We are looking for an experienced Test Manager to lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services.

This role combines hands‑on technical leadership with test strategy ownership. You will be responsible for defining and driving the overall testing approach across automated pipelines, data workflows, cloud infrastructure, and release delivery while managing risks, improving testing standards, and mentoring engineering teams.

The ideal candidate will have strong experience in Big Data and Data Lake environments, including testing both structured and unstructured datasets, and will be comfortable operating in highly automated CI/CD‑driven delivery environments.

You will work closely with engineering, DevOps, architecture, and stakeholder teams to ensure quality is embedded throughout the software delivery lifecycle.

Key Responsibilities

Define, implement, and manage the overall test strategy across cloud-native data platforms and applications

Lead end-to-end testing activities across data pipelines, cloud services, APIs, and distributed systems

Establish quality standards, governance processes, and testing best practices across teams

Drive a quality-first engineering culture across the programme

Provide leadership, mentoring, and coaching to Test Engineers and QA team members

Automated Testing & Frameworks

Design, implement, and oversee scalable automated test frameworks for data and application testing

Build and maintain automated test suites across AWS cloud infrastructure and data workflows

Enhance existing automation frameworks to improve reliability, coverage, maintainability, and reporting

Ensure automated testing is fully integrated within GitLab CI/CD pipelines for rapid feedback and release confidence

Big Data & Data Lake Testing

Lead testing activities across Big Data and Data Lake solutions containing structured and unstructured datasets

Validate large-scale data ingestion, transformation, and processing workflows

Ensure data integrity, transformation accuracy, reconciliation, and performance under load

Oversee PySpark-based data validation and interrogation activities on AWS EMR

Support testing and monitoring of workflows orchestrated through Apache Airflow

Drive root cause analysis for production data issues and platform defects

Identify, assess, and communicate quality risks across releases and environments

Analyse test execution metrics, defects, trends, and quality indicators

Translate technical findings into clear, prioritised reporting for technical and non-technical stakeholders

Provide regular status reporting to programme leadership and client stakeholders

CI/CD & DevOps

Collaborate with DevOps and engineering teams to improve release quality and deployment confidence

Ensure automated test execution within GitLab CI/CD pipelines

Support infrastructure testing and validation within Terraform-managed cloud environments

Contribute to continuous improvement initiatives across testing and delivery processes

Apache Airflow, Amazon Athena, AWS S3, AWS Glue, AWS EMR, AWS EC2, AWS ECR, AWS DynamoDB, AWS CloudWatch, AWS IAM, Python, SQL, Java, Bash, GitLab CI/CD, Jupyter Notebooks, Apache Spark / PySpark, Terraform, Docker

Key Skills

Strong proficiency in Python, SQL, Java, and scripting languages such as Bash

Hands‑on experience testing Big Data platforms and Data Lake solutions

Strong experience with AWS EMR, Apache Airflow, and Jupyter Notebooks

Experience performing PySpark-based data interrogation, validation, and testing

Strong knowledge of AWS services including S3, Glue, Athena, EMR, CloudWatch, IAM, DynamoDB, EC2, and ECR

Experience designing, implementing, and managing enterprise-scale automated test frameworks

Strong understanding of CI/CD practices using GitLab pipelines

Experience with infrastructure-as-code tools such as Terraform

Familiarity with Docker and containerised testing environments

Understanding of dimensional data models and slowly changing dimensions

Experience validating structured and unstructured data workflows

Strong analytical and root cause analysis skills

Ability to communicate technical risks clearly to stakeholders at all levels

Experience leading and mentoring QA/Test Engineering teams

Skills & Experience Required

Proven experience as aTest Manager, Lead Test Engineer, or QA Lead within cloud and data engineering environments

Strong background in automated testing for cloud-native and data-centric platforms

Experience testing large-scale data pipelines, transformation processes, and distributed systems

Experience managing or leading QA/testing teams in Agile delivery environments

Strong stakeholder management and communication skills

Experience establishing test governance, reporting, and quality metrics

Strong problem-solving and decision-making capability

Experience working within regulated or security-conscious delivery environments

SFIA Level 5 capability — provides leadership, influences strategy, and drives quality improvements

Active SC clearance (mandatory)

Diversity and Inclusion

AtScrumconnect Consulting , we believe diversity drives innovation and stronger outcomes for our teams and clients. We are committed to creating an inclusive environment where every individual is respected, valued, and supported. We welcome applications from candidates of all backgrounds, experiences, and perspectives.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Test Engineer (SC Cleared)
Test Engineer (SC Cleared)

Scrumconnect • Newcastle upon Tyne

Remote
GBP 50,000 - 70,000
Data Test Engineer (SC Cleared)
Data Test Engineer (SC Cleared)

Scrumconnect Ltd • Newcastle upon Tyne

On-site
GBP 90,000 - 130,000
Data Engineer
Data Engineer

Scrumconnect Ltd • Newcastle upon Tyne

Hybrid
GBP 70,000 - 95,000
Senior Data QA & Test Manager — Cloud Data & Automation
Senior Data QA & Test Manager — Cloud Data & Automation

Scrumconnect Ltd • Newcastle upon Tyne

On-site
GBP 90,000 - 130,000
Lead Performance Tester ( SC Cleared/ SC Eligible )
Lead Performance Tester ( SC Cleared/ SC Eligible )

Scrumconnect • Greater London

On-site
GBP 85,000 - 120,000
Cloud Data QA Lead — Big Data & Automation
Cloud Data QA Lead — Big Data & Automation

Scrumconnect • United Kingdom

Remote
GBP 90,000 - 140,000
Senior Software Engineer in Test, Data / ETL (Remote - EMEA)
Senior Software Engineer in Test, Data / ETL (Remote - EMEA)

Jobgether • United Kingdom

Remote
GBP 61,419 - 83,354
Unlimited paid time off
Comprehensive health insurance
Professional development stipend
Test Engineer
Test Engineer

Scrumconnect • United Kingdom

Hybrid
GBP 50,000 - 55,000
Test Engineer
Test Engineer

Response Informatics • Glasgow

On-site
GBP 45,000 - 65,000
QA and Test Engineer
QA and Test Engineer

Reed Professional Services LLP • Greater London

Hybrid
GBP 74,000 - 120,000
Hybrid work arrangement