Data Engineer

Department for Work and Pensions (DWP); Deutschland; Hybrid; EUR 36.000 - 60.000; Vollzeit

Benefits dieser Stelle

Time off volunteering

DWP 'I Can Be Me' program

Discounts and savings on shopping

Interest-free travel loans

Professional development & mentoring

Zusammenfassung

DWP is seeking a Data Engineer to join the Management Information team. You will develop PySpark-based ETL pipelines in an AWS-heavy hybrid environment and collaborate across stakeholders to deliver scalable data products for reporting and analytics.

Role requires experience with PySpark, AWS Lambda, and CI/CD, plus a strong full-stack background. Hybrid working model with competitive pay and a strong civil service pension.

Qualifikationen

• Experience developing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven data processing.
• Experience provisioning cloud infrastructure using EC2 and automation tools.
• Understanding of Agile practices and CI/CD delivery.
• Ability to optimise data processing, storage and queries for quality.
• Experience designing ETL pipelines with PySpark and Spark SQL.
• Strong full-stack background with C#, JavaScript, C++, Java

Aufgaben

• Deliver scalable data engineering solutions for analytical reporting and BI.
• Develop and orchestrate PySpark-based ETL pipelines in AWS.
• Collaborate with stakeholders to translate reporting requirements into robust solutions.
• Contribute to Agile delivery, CI/CD automation, and knowledge sharing.

Kenntnisse

PySpark workflows

AWS Lambda

CI/CD delivery

Data processing optimization

ETL pipelines (PySpark)

Full stack development (C#, JavaScript

Java (and C++/Java)

Tools

Amazon EMR

EC2

Spark SQL

CI/CD tooling

Jobbeschreibung

DWP. Digital with Purpose. We need a Data Engineer to join our Management Information (MI) team and help shape the future of data at DWP. Working with modern AWS technologies, you'll develop scalable data solutions that power reporting, analytics, and strategic decision-making. Our DWP Digital teams use fresh ideas and leading-edge tech to build and maintain digital solutions that will be used by nearly every person in the UK, every day and at key moments in their lives. DWP is the UK's largest government department. We help people into work and make payments worth over £195bn a year to support and empower millions of people. The scale of what we do is extraordinary, and our purpose is unique. We'd love you to join us.

What skills, knowledge and experience will you need?

• Experience developing and managing PySpark workflows within AWS environments.
• Knowledge of AWS Lambda and event-driven solutions to support automated data processing.
• Experience provisioning and supporting cloud infrastructure using Amazon EC2 and automation tools.
• Understanding of Agile ways of working and experience contributing to CI/CD delivery practices.
• Ability to optimise data processing, storage, and queries to improve performance and data quality.
• Experience designing and automating ETL pipelines using PySpark and Spark SQL.
• A strong full stack development background with technologies that include .NET 4,8, C#, JavaScript, C++ and Java.

You and your role As a Data Engineer in DWP Digital's Management Information (MI) team, you'll help us to deliver modern data engineering solutions that support analytical reporting, business intelligence, and strategic decision-making. We work in a hybrid cloud environment that combines AWS technologies with legacy platforms - here you will design, build, maintain, and optimise scalable data products used by our analysts and data scientists. You will develop and orchestrate PySpark-based ETL pipelines, using technologies such as Amazon EMR, AWS Lambda and EC2 to process large volumes of structured and unstructured data efficiently. Your role involves collaborating with a wide range of stakeholders to translate reporting requirements into robust technical solutions while ensuring high standards of data quality, performance, and security. Alongside hands-on data engineering delivery, you will contribute to Agile delivery practices, support CI/CD automation, maintain technical documentation, conduct code reviews, and share knowledge with fellow engineers and data professionals as part of a thriving community of practice.

Hybrid Working: We work a hybrid model - you'll spend some time working at home (40%) and some time collaborating face to face in a hub (60%).

Pay: We offer competitive pay for this role of up to £51,528.

Pension: You'll get a brilliant civil service pension with employer contributions of 28.97%, worth over £12,876 per year.

Holidays: A generous leave package starting at 26 days rising to 31 days over time. You can also take up to 3 extra days off a month on flexi-time. You'll also get all the usual public holidays.

Benefits

• Time off volunteering and charitable giving
• Bring your authentic self to work with 'I Can Be Me in DWP'
• Discounts and savings on shopping, fun days out and more
• Interest-free loans to buy a bike or a season ticket, so it's even easier for you to get to work and start making a difference
• Professional development, coaching, mentoring and career progression opportunities.

Awards and Recognition

• DWP have been recognised as 2024 Diversity Employer of the Year at the Computing Women in Tech Excellence awards
• Diverse and Inclusive Leadership at Digital Leaders Awards 2024
• Commended as Best Place to Work in Digital category in the Computing Digital Technology Leaders awards 2025
• Recognised as one of the Best Public Sector Employers at 2025 Women In Tech Employer Awards