Hybrid AWS Data Engineer: PySpark, ETL & Analytics

Manchester Digital; Manchester; Hybrid; GBP 42,000 - 52,000; Full time

Benefits offered by this job

Hybrid working model

Competitive pay up to £51,528

Job summary

DWP Digital in the United Kingdom is hiring a Data Engineer to join the Management Information team and help shape the future of data. You'll develop scalable data solutions on AWS that power reporting, analytics, and strategic decision‑making.

You will design and orchestrate PySpark ETL pipelines using EMR, Lambda, and EC2 in a hybrid cloud setup, collaborate with stakeholders, uphold data quality and security, and support CI/CD delivery and documentation.

Qualifications

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions for automated data processing.
• Experience provisioning and supporting cloud infrastructure using EC2 and automation tools.
• Understanding of Agile ways of working and CI/CD delivery practices.
• Ability to optimise data processing, storage and queries for performance and quality.
• Experience designing and automating PySpark ETL pipelines; strong full stack background with .NET 4/8, C#, JavaScript, C++, Java.

Responsibilities

• Deliver modern data engineering solutions supporting analytical reporting, BI and strategic decision‑making.
• Design, build, and optimise scalable data products in a hybrid cloud environment (AWS + legacy).
• Develop PySpark-based ETL pipelines using EMR, Lambda, and EC2 for large data volumes.
• Collaborate with stakeholders to translate reporting needs into robust technical solutions with data quality and security.

Skills

PySpark

AWS Lambda

Amazon EC2

CI/CD

ETL pipelines

Spark SQL

.NET

C#

JavaScript

C++

Java

Tools

Apache Spark

Job description

DWP Digital in the United Kingdom is hiring a Data Engineer to join the Management Information team and help shape the future of data. You'll develop scalable data solutions on AWS that power reporting, analytics, and strategic decision‑making.

You will design and orchestrate PySpark ETL pipelines using EMR, Lambda, and EC2 in a hybrid cloud setup, collaborate with stakeholders, uphold data quality and security, and support CI/CD delivery and documentation.