Data Engineer

Your World Recruitment Group; Cardiff; On-site; GBP 36,000 - 39,000; Full time

Job summary

Public Health Wales is seeking an Advanced Data Engineer to drive the modernization of national disease surveillance data platforms, migrating from on-premises to Google Cloud Platform to support CDSC. You will design, build, and optimize scalable ETL/ELT pipelines, apply DataOps, and ensure compliance with UK GDPR and NHS data governance while collaborating with stakeholders in an Agile environment.

You'll work with Spark, BigQuery, Airflow, and Docker in a cloud-first setup within Public

Qualifications

• Strong proficiency in SQL and Python; R desirable.
• Experience with cloud-native solutions (GCP), migrating on-prem to the cloud.
• Experience with big data processing (Spark, Dataproc, BigQuery) and orchestration (Airflow, Cloud Composer, Prefect).
• DevOps with Git, Docker, CI/CD and DataOps practices.
• Undergraduate degree in CS or related field.
• GCP Data Engineer or GCP Cloud Architect certification desirable.

Responsibilities

• Data Engineering & Cloud Architecture: Design, build, test, and maintain scalable ETL/ELT pipelines; migrate legacy data platforms to GCP.
• Pipeline Orchestration & Optimization: Monitor, troubleshoot, and optimize pipelines using Spark, BigQuery, Airflow, Cloud Composer, Prefect.
• Software & DataOps Best Practices: Apply CI/CD, containerization (Docker), Git-based version control, automated testing; maintain documentation.
• Data Modeling & Governance: Develop data models; implement metadata, data lineage, validation frameworks; ensure GDPR compliance.
• Collaboration & Delivery: Work in an Agile environment; translate business needs into robust technical solutions; communicate clearly with stakeholders.

Skills

SQL

Python

R (desirable)

Education

Bachelor's degree in Computer Science or related

Tools

Docker

Airflow

Cloud Composer

Prefect

BigQuery

Dataproc

Spark

Git

CI/CD

Job description

Duration of booking: Expected to last till March 2027 with possible extension

Proposed start date: ASAP

Pay Rates: Up to£28 per hour Umbrella or £26 per hour PAYE inclusive of holidaypay

Hours / Working Days: 37.5 hours per week / Monday to Friday, 9am – 5pm

Based: Office / Hospital

About the Role

We are seeking an Advanced Data Engineer to play a pivotal role in supporting the NHS transition to cloud computing, specifically driving data platform modernization for the Communicable Disease Surveillance Centre (CDSC) within Public Health Wales.

In this role, you will design, build, and optimize scalable data pipelines that support national disease surveillance and public health intelligence. You will lead the migration of mission-critical surveillance data platforms from on-premises infrastructure to Google Cloud Platform (GCP), ensuring seamless data orchestration, rigorous DataOps practices, and adherence to NHS data security and governance standards.

Key Responsibilities

• Data Engineering & Cloud Architecture: Design, build, test, and maintain scalable ETL/ELT pipelines and production-ready data products. Lead the migration of legacy data platforms, pipelines, and services from on-premises infrastructure to GCP.
• Pipeline Orchestration & Optimization: Monitor, troubleshoot, and optimize pipeline performance for large-scale datasets using distributed processing tools (Spark, BigQuery) and workflow orchestrators (Airflow, Cloud Composer, Dagster).
• Software & DataOps Best Practices: Apply CI/CD, containerization (Docker), Git-based version control, and automated testing to build production-quality code. Conduct peer code reviews, refactor existing code, and maintain robust technical documentation.
• Data Modeling & Governance: Develop conceptual, logical, and physical data models. Implement metadata management, data lineage, data profiling, and validation frameworks to maintain high data quality. Ensure full compliance with UK GDPR, the Data Protection Act 2018, and ethical data guidelines.
• Collaboration & Delivery: Work in an Agile environment to translate non-technical business requirements into robust technical solutions. Communicate architectural and data concepts clearly to both technical peers and key business stakeholders.

Requirements & Qualifications

• Technical Core: Strong proficiency in SQL and Python; working knowledge or experience with R is highly desirable.
• Cloud & Infrastructure: Hands-on experience designing cloud-native solutions (preferably GCP), migrating on-premise systems to the cloud, and managing relational databases/data warehouses.
• Data Engineering Stack: Demonstrated experience with big data processing (Spark, Dataproc, BigQuery), orchestration (Airflow/Cloud Composer/Prefect), and API-driven data integration.
• DevOps & Quality: Proven background with Git (GitHub/GitLab/Azure DevOps), Docker, CI/CD automation, and modern DataOps principles.
• Education & Certifications:
• Undergraduate degree in Computer Science, a related technical field, or equivalent practical work experience.
• GCP Data Engineer or GCP Cloud Architect certification is highly desirable.