Hybrid Data Engineer: PySpark & AWS for Scalable Analytics

Allscreens Nationwide Ltd; Leeds; On-site; GBP 31,000 - 52,000; Full time

Benefits offered by this job

Hybrid working

Civil service pension

Generous holidays

Job summary

DWP Digital's MI team is looking for a Data Engineer to shape the future of data at DWP. You will work with modern AWS technologies to develop scalable data solutions powering reporting, analytics and strategic decision-making.

You will build PySpark-based ETL pipelines on EMR, Lambda and EC2, collaborating with stakeholders to translate reporting requirements into robust data solutions. Hybrid working model with time split between home and hub is supported.

Qualifications

• Experience developing and managing PySpark workflows within AWS.
• Knowledge of AWS Lambda and event-driven data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2.
• Understanding of Agile ways of working and CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries for performance and quality.

Responsibilities

• Deliver modern data engineering solutions for analytical reporting, BI and decision-making.
• Design, build, maintain and optimise scalable data products used by analysts and data scientists.
• Develop PySpark-based ETL pipelines using EMR, Lambda, and EC2.
• Collaborate with stakeholders to translate reporting requirements into robust technical solutions.
• Contribute to Agile delivery, CI/CD automation, documentation and knowledge sharing.

Skills

PySpark

AWS Lambda

EC2

Agile & CI/CD

Data processing optimization

Tools

PySpark

AWS Lambda

Amazon EMR

Job description

DWP Digital's MI team is looking for a Data Engineer to shape the future of data at DWP. You will work with modern AWS technologies to develop scalable data solutions powering reporting, analytics and strategic decision-making.

You will build PySpark-based ETL pipelines on EMR, Lambda and EC2, collaborating with stakeholders to translate reporting requirements into robust data solutions. Hybrid working model with time split between home and hub is supported.