Data Engineer — PySpark & AWS Data Pipelines

Career Choices Dewis Gyrfa Ltd; Birmingham; Hybrid; GBP 46,000 - 57,000; Full time

Benefits offered by this job

Hybrid working model

Civil service pension

Job summary

DWP Digital is seeking a Data Engineer to join the Management Information (MI) team. You will build scalable data products, develop PySpark-based ETL pipelines with EMR, Lambda, and EC2, and collaborate across stakeholders to translate reporting needs into robust solutions.

You will contribute to Agile delivery, CI/CD automation, and maintain documentation while ensuring data quality, performance, and security in a hybrid cloud environment.

Qualifications

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
• Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries to improve performance and data quality.

Responsibilities

• Design, build, maintain, and optimise scalable data products used by analysts and data scientists.
• Develop and orchestrate PySpark-based ETL pipelines using technologies such as Amazon EMR, AWS Lambda, and EC2.
• Collaborate with a wide range of stakeholders to translate reporting requirements into robust technical solutions while ensuring data quality, performance, and security.
• Contribute to Agile delivery practices, support CI/CD automation, maintain technical documentation, and conduct code reviews.

Skills

PySpark workflows

AWS Lambda

CI/CD

Agile

Data quality

EC2 automation

Tools

Amazon EMR

AWS EC2

Automation tooling

Job description

DWP Digital is seeking a Data Engineer to join the Management Information (MI) team. You will build scalable data products, develop PySpark-based ETL pipelines with EMR, Lambda, and EC2, and collaborate across stakeholders to translate reporting needs into robust solutions.

You will contribute to Agile delivery, CI/CD automation, and maintain documentation while ensuring data quality, performance, and security in a hybrid cloud environment.