SW Secure Data Environment Senior Data Engineer

SW Secure Data Environment Senior Data Engineer; West of England, Gloucester; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

Southwest Secure Data Environment (SWSDE) seeks an experienced Senior Data Engineer to provide technical leadership and hands-on expertise in designing, building and delivering secure health data pipelines and research-ready datasets.

You will work with NHS providers, researchers and delivery partners to translate needs into scalable, compliant engineering solutions, profiling data quality and guiding junior engineers across complex projects.

Qualifications

• Educated to master's level or equivalent senior-level experience within a specialist data, analytics, digital or technical environment.
• Significant experience designing, developing and supporting complex data pipelines, transformations and data integration solutions.
• Strong hands-on experience using SQL, Python and relevant modern data engineering tools and technologies.
• Experience working with large-scale health, research, public sector or similarly complex datasets.
• Strong knowledge of data modelling, data quality assurance, metadata management and data validation techniques.
• Ability to analyse complex datasets, data structures and technical requirements and determine appropriate engineering approaches.
• Experience planning and delivering complex data engineering or integration activity within agreed timescales and resources.
• Ability to manage multiple projects, technical priorities and deliverables simultaneously.
• Experience providing technical leadership, peer review, mentoring and guidance to engineers or technical colleagues.
• Excellent communication skills, with the ability to translate complex requirements and explain technical concepts to technical and non-technical audiences.
• Strong problem-solving skills and the ability to resolve complex data quality, integration and interoperability issues.
• Understanding of information governance, cyber security, data protection and relevant compliance requirements.
• Ability to work collaboratively with data providers, researchers, suppliers and partner organisations across organisational boundaries.

Responsibilities

• Lead the design, development, testing and delivery of complex health and care data pipelines, models and research-ready datasets.
• Translate research, operational and provider needs into technical specifications and secure, scalable, reusable solutions using SQL, Python and modern engineering technologies.
• Work with NHS providers to understand source systems, datasets, coding, data flows and constraints.
• Profile and validate data, resolving quality issues and leading extraction, ingestion, transformation, linkage and curation.
• Automate processes to improve the reliability, repeatability, traceability and efficiency of data delivery.
• Coordinate engineering across projects, datasets and providers; identify risks, dependencies, capacity issues and barriers, acting or escalating as needed.
• Guide and mentor engineers, peer-review code, pipelines, models, transformations and documentation.
• Promote version control, testing, documentation, reusable components and agreed development practices.
• Work with platform, cloud, architecture, cyber security and information governance teams to meet requirements.
• Maintain documentation, metadata, data dictionaries, lineage and quality controls, and improve SWSDE standards, patterns and cloud capability.

Skills

SQL

Python

Data modelling

Data quality

Metadata management

Data validation

Leadership

Education

Master's degree in data/analytics/technical field

Tools

Azure

AWS

Git

Airflow

Job description

Job overview

We are seeking an experienced Senior Data Engineer to provide technical leadership and hands-on expertise within the Southwest Secure Data Environment (SWSDE). SWSDE enables secure use of health and care data for research, innovation, service improvement and transformation. Working with NHS organisations, universities and research partners, it supports better outcomes and data-driven decisions.

The postholder will lead the design, development, testing and implementation of complex data pipelines, models and research-ready datasets. Working with NHS providers, researchers, technical teams and delivery partners, you will turn complex requirements into secure, scalable and reusable engineering solutions.

You will combine strong technical skills with the ability to lead complex engineering activity, solve difficult data problems and develop other engineers. Essential experience includes SQL, Python, data integration, data modelling and modern engineering practices.

NHS or healthcare data experience is valuable, but we also welcome applicants from research, public-sector or other complex, regulated environments who offer strong engineering skills and fresh thinking.

The role is administratively based in Bristol, with applications welcome from across the Southwest. Flexible regional and home working is available, with travel to partner organisations as required.

Advert

The postholder will:

• Lead the design, development, testing and delivery of complex health and care data pipelines, models and research-ready datasets.
• Translate research, operational and provider needs into technical specifications and secure, scalable, reusable solutions using SQL, Python and modern engineering technologies.
• Work with NHS providers to understand source systems, datasets, coding, data flows and constraints.
• Profile and validate data, resolving quality issues and leading extraction, ingestion, transformation, linkage and curation.
• Automate processes to improve the reliability, repeatability, traceability and efficiency of data delivery.
• Coordinate engineering across projects, datasets and providers; identify risks, dependencies, capacity issues and barriers, acting or escalating as needed.
• Guide and mentor engineers, peer-review code, pipelines, models, transformations and documentation.
• Promote version control, testing, documentation, reusable components and agreed development practices.
• Work with platform, cloud, architecture, cyber security and information governance teams to meet requirements.
• Maintain documentation, metadata, data dictionaries, lineage and quality controls, and improve SWSDE standards, patterns and cloud capability.

Working for our organisation

We are committed to creating an inclusive organisation that promotes and values diversity. We know from experience that different ideas, perspectives and backgrounds create a stronger, more creative workplace that helps us to deliver the best services. We welcome applications which represent the rich diversity of our community; from people of all ages, disabled people, all genders, people from ethnic minority groups, LGBTQ+ people, people with diverse gender identity and expression, people from all religions and beliefs and other diverse characteristics, and we have processes in place to ensure that all applications are treated fairly and consistently at every stage of the recruitment process.

Detailed job description and main responsibilities

Essential

• Educated to master's level or equivalent senior-level experience within a specialist data, analytics, digital or technical environment.
• Significant experience designing, developing and supporting complex data pipelines, transformations and data integration solutions.
• Strong hands-on experience using SQL, Python and relevant modern data engineering tools and technologies.
• Experience working with large-scale health, research, public sector or similarly complex datasets.
• Strong knowledge of data modelling, data quality assurance, metadata management and data validation techniques.
• Ability to analyse complex datasets, data structures and technical requirements and determine appropriate engineering approaches.
• Experience planning and delivering complex data engineering or integration activity within agreed timescales and resources.
• Ability to manage multiple projects, technical priorities and deliverables simultaneously.
• Experience providing technical leadership, peer review, mentoring and guidance to engineers or technical colleagues.
• Excellent communication skills, with the ability to translate complex requirements and explain technical concepts to technical and non-technical audiences.
• Strong problem-solving skills and the ability to resolve complex data quality, integration and interoperability issues.
• Understanding of information governance, cyber security, data protection and relevant compliance requirements.
• Ability to work collaboratively with data providers, researchers, suppliers and partner organisations across organisational boundaries.
• Commitment to technical quality, inclusive working, knowledge sharing, innovation and continuous improvement.

Desirable

• Experience working with NHS data, health research infrastructure, Secure Data Environments or Trusted Research Environments.
• Knowledge of NHS datasets, coding systems, data standards and healthcare information flows.
• Experience of cloud-based data engineering using Azure, AWS or an equivalent platform.
• Experience of DevOps, automation, infrastructure-as-code or platform engineering approaches.
• Knowledge of interoperability standards or common data models such as OMOP, FHIR or openEHR.
• Knowledge of SATRE, the Data Security and Protection Toolkit, ISO 27001 or other relevant security and assurance frameworks.