Senior Data Engineer: Azure, PySpark & Pipelines | Hybrid

DWP; United Kingdom; Hybrid; GBP 68,000 - 83,000; Full time

Benefits offered by this job

Flexible working hours

Volunteer time off

I Can Be Me in DWP

Discounts and savings

Interest-free travel loans

Sports and social activities

Professional development and mentoring

Job summary

DWP (Department for Work and Pensions) is hiring a Data Engineer to design and deliver cloud-based data solutions. You will lead scalable ETL/ELT pipelines using Python and PySpark, ensuring trusted, high-quality data for decision-making, while guiding teams and promoting engineering standards.

Hybrid working and flexible hours are available across multiple UK locations, with competitive pay and a generous pension.

Qualifications

• Experience designing and delivering complex ETL/ELT pipelines.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences.
• Strong understanding of cloud and distributed data processing.
• Ability to analyse complex requirements and define testing approaches.
• Experience leading resolution of complex data, platform and pipeline issues.
• Knowledge migrating and transforming data from legacy/on-premise systems to cloud.
• Expertise automating ETL/ELT with PySpark and Spark SQL on Azure data environments.

Responsibilities

• Design, build and enhance scalable ETL/ELT pipelines using Python and PySpark.
• Collaborate with stakeholders to translate complex business needs into data solutions.
• Manage data across multiple platforms, ensuring security, accessibility and quality.
• Lead teams through large-scale data/platform transformations and promote best practices.

Skills

ETL pipelines

Stakeholder communication

Cloud data

Requirements analysis

Issue resolution

Data migration

PySpark / Spark SQL

Tools

Azure

Job description

DWP (Department for Work and Pensions) is hiring a Data Engineer to design and deliver cloud-based data solutions. You will lead scalable ETL/ELT pipelines using Python and PySpark, ensuring trusted, high-quality data for decision-making, while guiding teams and promoting engineering standards.

Hybrid working and flexible hours are available across multiple UK locations, with competitive pay and a generous pension.