Data Engineer — AWS PySpark ETL (Hybrid)

DWP Digital; Lancashire; Hybrid; GBP 31,000 - 52,000; Full time

Benefits offered by this job

Hybrid working model (40% home, 60% in

Civil service pension 28.97%

26–31 days holiday + flexi-time

Professional development

Travel/commute assistance

Job summary

DWP Digital, part of the UK Civil Service, is recruiting a Data Engineer to join the Management Information (MI) team. You will design, build and optimise scalable data products using AWS, PySpark and EMR to support analytics and reporting for government decisions.

You will work in a hybrid cloud environment, delivering end-to-end data pipelines and contributing to Agile delivery, CI/CD practices and high data quality across large datasets.

Qualifications

• Experience developing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions for automated data processing.
• Experience provisioning cloud infrastructure using EC2 and automation tools.

Responsibilities

• Develop and orchestrate PySpark-based ETL pipelines using AWS technologies (EMR, Lambda, EC2).
• Translate reporting requirements into robust technical solutions with high data quality, performance and security.
• Collaborate with stakeholders to deliver analytical reporting, BI and strategic decision-making.
• Contribute to Agile delivery practices and support CI/CD automation.

Skills

PySpark workflows

AWS Lambda

EC2

CI/CD

Agile

Data quality

Tools

Amazon EMR

Job description

DWP Digital, part of the UK Civil Service, is recruiting a Data Engineer to join the Management Information (MI) team. You will design, build and optimise scalable data products using AWS, PySpark and EMR to support analytics and reporting for government decisions.

You will work in a hybrid cloud environment, delivering end-to-end data pipelines and contributing to Agile delivery, CI/CD practices and high data quality across large datasets.