Senior Data Engineer - Secure Health Data Pipelines Lead

SW Secure Data Environment Senior Data Engineer; West of England, Gloucester; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Southwest Secure Data Environment (SWSDE) seeks an experienced Senior Data Engineer to provide technical leadership and hands-on expertise in designing, building and delivering secure health data pipelines and research-ready datasets.

You will work with NHS providers, researchers and delivery partners to translate needs into scalable, compliant engineering solutions, profiling data quality and guiding junior engineers across complex projects.

Qualifications

• Educated to master's level or equivalent senior-level experience within a specialist data, analytics, digital or technical environment.
• Significant experience designing, developing and supporting complex data pipelines, transformations and data integration solutions.
• Strong hands-on experience using SQL, Python and relevant modern data engineering tools and technologies.
• Experience working with large-scale health, research, public sector or similarly complex datasets.
• Strong knowledge of data modelling, data quality assurance, metadata management and data validation techniques.
• Ability to analyse complex datasets, data structures and technical requirements and determine appropriate engineering approaches.
• Experience planning and delivering complex data engineering or integration activity within agreed timescales and resources.
• Ability to manage multiple projects, technical priorities and deliverables simultaneously.
• Experience providing technical leadership, peer review, mentoring and guidance to engineers or technical colleagues.
• Excellent communication skills, with the ability to translate complex requirements and explain technical concepts to technical and non-technical audiences.
• Strong problem-solving skills and the ability to resolve complex data quality, integration and interoperability issues.
• Understanding of information governance, cyber security, data protection and relevant compliance requirements.
• Ability to work collaboratively with data providers, researchers, suppliers and partner organisations across organisational boundaries.

Responsibilities

• Lead the design, development, testing and delivery of complex health and care data pipelines, models and research-ready datasets.
• Translate research, operational and provider needs into technical specifications and secure, scalable, reusable solutions using SQL, Python and modern engineering technologies.
• Work with NHS providers to understand source systems, datasets, coding, data flows and constraints.
• Profile and validate data, resolving quality issues and leading extraction, ingestion, transformation, linkage and curation.
• Automate processes to improve the reliability, repeatability, traceability and efficiency of data delivery.
• Coordinate engineering across projects, datasets and providers; identify risks, dependencies, capacity issues and barriers, acting or escalating as needed.
• Guide and mentor engineers, peer-review code, pipelines, models, transformations and documentation.
• Promote version control, testing, documentation, reusable components and agreed development practices.
• Work with platform, cloud, architecture, cyber security and information governance teams to meet requirements.
• Maintain documentation, metadata, data dictionaries, lineage and quality controls, and improve SWSDE standards, patterns and cloud capability.

Skills

SQL

Python

Data modelling

Data quality

Metadata management

Data validation

Leadership

Education

Master's degree in data/analytics/technical field

Tools

Azure

AWS

Git

Airflow

Job description

Southwest Secure Data Environment (SWSDE) seeks an experienced Senior Data Engineer to provide technical leadership and hands-on expertise in designing, building and delivering secure health data pipelines and research-ready datasets.

You will work with NHS providers, researchers and delivery partners to translate needs into scalable, compliant engineering solutions, profiling data quality and guiding junior engineers across complex projects.