Data Engineer: PySpark on AWS for Scalable Analytics

Allscreens Nationwide Ltd; Sheffield; Hybrid; GBP 46,000 - 57,000; Full time

Benefits offered by this job

Hybrid working (40% home / 60% hub)

Civil service pension 28.97%

Holidays 26–31 days + flexi-time

Job summary

DWP Digital's Management Information (MI) team is seeking a Data Engineer to shape data at scale using AWS technologies, PySpark, and ETL pipelines for reporting, analytics, and strategic decision-making.

Join a hybrid environment combining AWS with legacy platforms to build scalable data products used by analysts and data scientists. You will orchestrate PySpark ETL pipelines with EMR, Lambda, and EC2, while collaborating with stakeholders and supporting CI/CD automation.

Qualifications

• Experience with PySpark workflows in AWS environments.
• Knowledge of AWS Lambda and event-driven data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2.

Responsibilities

• Deliver modern data engineering solutions to support analytical reporting and BI.
• Collaborate with stakeholders to translate reporting needs into robust technical solutions.
• Contribute to Agile delivery practices and CI/CD automation; maintain technical docs and perform code reviews.
• Share knowledge with engineers and data professionals as part of a community of practice.

Skills

PySpark

AWS Lambda

EC2

CI/CD

Agile

Data optimization

Tools

Amazon EMR

EC2

CI/CD tooling

Job description

DWP Digital's Management Information (MI) team is seeking a Data Engineer to shape data at scale using AWS technologies, PySpark, and ETL pipelines for reporting, analytics, and strategic decision-making.

Join a hybrid environment combining AWS with legacy platforms to build scalable data products used by analysts and data scientists. You will orchestrate PySpark ETL pipelines with EMR, Lambda, and EC2, while collaborating with stakeholders and supporting CI/CD automation.