Senior Data Engineer — Azure Pipelines, Hybrid

DWP; Stafford; Hybrid; GBP 68,000 - 83,000; Full time

Benefits offered by this job

Flexible working

Volunteer time

I Can Be Me in DWP

Discounts and savings

Interest-free travel loans

Job summary

DWP is seeking an experienced Data Engineer to design, build and enhance scalable ETL/ELT pipelines using Python and PySpark in Azure. You will lead the design and delivery of modern cloud-based data solutions, ensuring high-quality, trusted data for decision-making across the organisation.

You will work with stakeholders to translate complex requirements into efficient data models, while mentoring the team and promoting engineering best practices.

Qualifications

• Experience designing and delivering complex ETL/ELT pipelines.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences.
• Strong understanding of cloud and distributed data processing for scalable data solutions.
• Proven ability to analyse requirements and define testing approaches.
• Experience leading resolution of data, platform and pipeline issues.
• Knowledge migrating data from legacy/on-premise systems to cloud platforms.
• Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL in Azure environments.

Responsibilities

• Design, build and enhance scalable ETL/ELT pipelines using Python and PySpark.
• Lead the design and delivery of modern cloud-based data solutions across teams.
• Collaborate with stakeholders to translate complex requirements into trusted datasets.
• Coach and develop team members while driving engineering best practices.

Skills

ETL/ELT pipelines

PySpark

Python

Azure

Data governance

Stakeholder communication

Cloud data processing

Tools

Azure Data Factory

Spark SQL

Job description

DWP is seeking an experienced Data Engineer to design, build and enhance scalable ETL/ELT pipelines using Python and PySpark in Azure. You will lead the design and delivery of modern cloud-based data solutions, ensuring high-quality, trusted data for decision-making across the organisation.

You will work with stakeholders to translate complex requirements into efficient data models, while mentoring the team and promoting engineering best practices.