Data Engineer — PySpark & AWS for Strategic Analytics

Allscreens Nationwide Ltd; Lancashire; Hybrid; GBP 31,000 - 52,000; Full time

Benefits offered by this job

Hybrid working (40% home, 60% hub)

Pension with employer contributions

Generous holidays

Professional development

Job summary

DWP Digital is seeking a Data Engineer to join the Management Information (MI) team. You will build scalable data solutions in a hybrid cloud environment using AWS and PySpark, delivering analytics-ready data for reporting and decision-making.

You will collaborate with analysts and data scientists, contribute to CI/CD, and help ensure data quality and security while working in a flexible hybrid setup with 40% home and 60% hub time.

Qualifications

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
• Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries to improve performance and data quality.

Responsibilities

• As a Data Engineer in DWP Digital's MI team, you'll help deliver modern data engineering solutions that support analytical reporting, BI, and strategic decision-making.
• Design, build, maintain, and optimise scalable data products used by analysts and data scientists in a hybrid cloud environment.
• Develop and orchestrate PySpark-based ETL pipelines using AWS services to process large volumes of data.
• Collaborate with stakeholders to translate reporting requirements into robust technical solutions with high data quality, performance and security.
• Contribute to Agile delivery, CI/CD automation, documentation, code reviews, and knowledge sharing.

Skills

PySpark

AWS Lambda

EC2

Agile

CI/CD

Data quality

Tools

AWS

EMR

Python

Job description

DWP Digital is seeking a Data Engineer to join the Management Information (MI) team. You will build scalable data solutions in a hybrid cloud environment using AWS and PySpark, delivering analytics-ready data for reporting and decision-making.

You will collaborate with analysts and data scientists, contribute to CI/CD, and help ensure data quality and security while working in a flexible hybrid setup with 40% home and 60% hub time.