Hybrid Data Engineer — PySpark & AWS for Analytics

DWP Digital; Manchester; Hybrid; GBP 39,000 - 52,000; Full time

Benefits offered by this job

Hybrid working model

Civil service pension with employer 28

Holidays 26–31 days

Job summary

DWP Digital in the United Kingdom is seeking a Data Engineer to join the Management Information team. You will design and implement scalable data pipelines in AWS, supporting reporting, analytics and strategic decision-making for the department.

You will work with PySpark ETL on EMR, Lambda and EC2, collaborate with stakeholders, and contribute to Agile delivery and CI/CD in a hybrid working pattern.

Qualifications

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
• Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries to improve performance and data quality.

Responsibilities

• As a Data Engineer in DWP Digital's Management Information (MI) team, deliver modern data engineering solutions that support analytical reporting, business intelligence, and strategic decision-making.
• Design, build, maintain, and optimise scalable data products used by analysts and data scientists in a hybrid cloud environment.
• Develop and orchestrate PySpark-based ETL pipelines using AWS EMR, Lambda, and EC2 to process large volumes of data efficiently.
• Collaborate with stakeholders to translate reporting requirements into robust technical solutions with high data quality, performance and security.
• Contribute to Agile delivery, support CI/CD automation, maintain technical documentation, conduct code reviews, and share knowledge within the engineering community.

Skills

PySpark

AWS Lambda

CI/CD

Agile delivery

Data optimization

Tools

Amazon EMR

Amazon EC2

Job description

DWP Digital in the United Kingdom is seeking a Data Engineer to join the Management Information team. You will design and implement scalable data pipelines in AWS, supporting reporting, analytics and strategic decision-making for the department.

You will work with PySpark ETL on EMR, Lambda and EC2, collaborate with stakeholders, and contribute to Agile delivery and CI/CD in a hybrid working pattern.