Senior Data Engineer - Azure ETL Leader

DWP Digital; Newcastle upon Tyne; Hybrid; GBP 68,000 - 76,000; Full time

Benefits offered by this job

Hybrid working

Civil service pension

Generous leave

Professional development

Job summary

DWP Digital is seeking a Senior Data Engineer to design, build and optimise cloud-based data pipelines using PySpark and Spark SQL in Azure. You will lead technical teams and collaborate with stakeholders to deliver scalable, trusted data solutions that support decision making across the organisation.

Hybrid working model in the UK, with opportunities across multiple sites, including Newcastle upon Tyne, Manchester and London.

Qualifications

• Proven experience designing and delivering complex ETL/ELT pipelines.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences.
• Strong understanding of cloud and distributed data processing, designing scalable, resilient data solutions.
• Proven ability to analyse complex requirements and define testing approaches.
• Experience leading resolution of complex data, platform and pipeline issues across levels.
• Strong knowledge of migrating data from legacy/on-premise to cloud platforms.
• Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL in Azure environments.

Responsibilities

• Design, build and enhance scalable ETL/ELT pipelines using Python and PySpark.
• Transform data into trusted datasets for insights, reporting and decision-making.
• Manage and optimise data across platforms ensuring security and accessibility.
• Lead teams through large-scale data and platform transformation and set engineering direction.
• Drive automation, platform performance and governance to deliver high-quality data services.

Skills

ETL/ELT pipelines

PySpark

Spark SQL

Cloud data processing

Stakeholder communication

Team leadership

Tools

Azure

Job description

DWP Digital is seeking a Senior Data Engineer to design, build and optimise cloud-based data pipelines using PySpark and Spark SQL in Azure. You will lead technical teams and collaborate with stakeholders to deliver scalable, trusted data solutions that support decision making across the organisation.

Hybrid working model in the UK, with opportunities across multiple sites, including Newcastle upon Tyne, Manchester and London.