Senior Data Engineer — Azure, PySpark, Hybrid Leadership

DWP Digital; West End; Hybrid; GBP 45,000 - 76,000; Full time

Benefits offered by this job

Flexible working hours

Hybrid working model

Pension contributions

Job summary

DWP Digital, part of the UK government, is hiring a Senior Data Engineer to design, build and lead scalable ETL/ELT pipelines in Azure. You will drive data solutions that support millions of people and guide teams across multiple locations.

The role focuses on cloud-based data platforms, data governance and delivering high-quality datasets for decision-making. Hybrid work and extensive development opportunities are offered.

Qualifications

• Proven experience designing and delivering complex ETL/ELT pipelines.
• Ability to communicate complex technical concepts to technical and non-technical audiences.
• Strong understanding of cloud and distributed data processing.
• Experience leading resolution of data and pipeline issues across levels.
• Knowledge of migrating data from legacy/on-prem systems to cloud platforms.

Responsibilities

• Design, build and enhance scalable ETL/ELT pipelines using Python and PySpark.
• Lead technical teams and shape technical direction and best practices.
• Ensure data solutions are reliable, secure and fit for purpose.
• Drive automation, platform performance and Agile delivery.
• Collaborate with stakeholders to transform complex business requirements into data models.

Skills

ETL/ELT pipelines

Azure

PySpark

Spark SQL

Data leadership

Tools

PySpark

Spark SQL

Azure

Job description

DWP Digital, part of the UK government, is hiring a Senior Data Engineer to design, build and lead scalable ETL/ELT pipelines in Azure. You will drive data solutions that support millions of people and guide teams across multiple locations.

The role focuses on cloud-based data platforms, data governance and delivering high-quality datasets for decision-making. Hybrid work and extensive development opportunities are offered.