Data Engineer – AWS PySpark & Scalable Data Pipelines

DWP Digital; Sheffield; Hybrid; GBP 31,000 - 52,000; Full time

Benefits offered by this job

Hybrid working

Competitive pay up to £51,528

Civil service pension 28.97%

Holidays 26–31 days + flexi-time

Broad benefits package

Job summary

DWP Digital is seeking a Data Engineer to join the Management Information team and help shape the future of data. You will develop scalable data solutions using AWS technologies to power reporting, analytics, and strategic decision-making.

The role operates in a hybrid cloud environment, designing and maintaining data products used by analysts and data scientists, and contributing to Agile and CI/CD practices to ensure high standards of data quality and security.

Qualifications

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
• Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries to improve performance and data quality.

Responsibilities

• As a Data Engineer in DWP Digital's MI team, you'll help deliver modern data engineering solutions that support analytical reporting, business intelligence, and strategic decision-making.
• Work in a hybrid cloud environment combining AWS technologies with legacy platforms to design, build, maintain, and optimise scalable data products used by analysts and data scientists.
• Develop and orchestrate PySpark-based ETL pipelines using Amazon EMR, AWS Lambda, and EC2 to process large volumes of structured and unstructured data efficiently.
• Collaborate with stakeholders to translate reporting requirements into robust technical solutions while ensuring data quality, performance, and security.
• Contribute to Agile delivery, support CI/CD automation, maintain technical documentation, conduct code reviews, and share knowledge within a community of practice.

Skills

PySpark workflows

AWS

Lambda

EC2

CI/CD

data quality

Tools

Amazon EMR

AWS Lambda

EC2

Job description

DWP Digital is seeking a Data Engineer to join the Management Information team and help shape the future of data. You will develop scalable data solutions using AWS technologies to power reporting, analytics, and strategic decision-making.

The role operates in a hybrid cloud environment, designing and maintaining data products used by analysts and data scientists, and contributing to Agile and CI/CD practices to ensure high standards of data quality and security.