Read more

DWP Digital; Coventry; Hybrid; GBP 45,000 - 76,000; Full time

Benefits offered by this job

Flexible hours

Hybrid working

Professional development and mentoring

Pension contributions (28.97%)

Discounts and savings

Job summary

DWP Digital in the United Kingdom is seeking a Senior Data Engineer to design, build and maintain cloud-based data pipelines using Python and PySpark. You will help ensure trusted data for decision-making and work with stakeholders across the organisation.

You will lead data engineering initiatives, mentor teammates and drive automation and governance while embracing Agile delivery and hybrid working arrangements.

Qualifications

• Proven experience designing and delivering complex ETL/ELT pipelines.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences, including senior stakeholders.
• Strong understanding of cloud and distributed data processing, designing scalable, resilient and maintainable data solutions.
• Proven ability to analyse complex requirements, produce clear specifications and define effective testing approaches.
• Experience leading the resolution of complex data, platform and pipeline issues across strategic, tactical and operational levels.
• Strong knowledge of migrating and transforming data from legacy and on-premise systems to cloud platforms.
• Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments.

Responsibilities

• Design, build and enhance scalable ETL/ELT pipelines using Python and PySpark, ensuring data solutions are reliable and aligned with engineering best practice.
• Work closely with stakeholders to understand complex business requirements and transform data into trusted datasets for insights.
• Manage and optimise data across multiple platforms, ensuring security, accessibility and fit for purpose.
• Lead teams through large-scale data and platform transformation and develop capability across the team.
• Develop complex data pipelines, drive automation and champion Agile delivery, data governance and operational excellence.

Skills

ETL/ELT pipeline design

PySpark

Spark SQL

Python

Cloud data engineering

Stakeholder communication

Leadership

Agile delivery

Tools

Azure

Spark

Job description

Pay of up to £75,664, 28.97% employer pension contributions, hybrid working, flexible hours, and great work life balance.

DWP. Digital with Purpose.

Are you an experienced Senior Data Engineer with strong skills in Azure and data pipelines and experience leading technical teams?

Join DWP's Data Engineering community and help build the data platforms and services that support millions of people across the UK.

You’ll lead the design and delivery of modern cloud-based data solutions, ensuring high-quality, trusted data is available to drive better decision-making across the organisation while supporting engineering standards and developing others within the team.

We have opportunities across three areas of DWP Digital:

Skills England - delivering data-driven insights and digital services to support skills development, workforce planning and technical education (London, Coventry, Darlington, Sheffield and Leeds).

Data & Analytics - within the Architecture and Engineering Technical Oversight Team, providing technical assurance and governance across data platforms, products and services (Blackpool, Manchester, Leeds and Newcastle).

Technology Services - delivering the secure cloud, connectivity, network and security foundations that underpin DWP’s digital services (Manchester, Blackpool, Newcastle, Leeds and Birmingham).

Successful candidates will be matched to the most suitable business area, with team and location details discussed during the recruitment process.

What skills, knowledge and experience will you need?

• Proven experience designing and delivering complex ETL/ELT pipelines, managing priorities and driving quality outcomes.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences, including senior stakeholders.
• Strong understanding of cloud and distributed data processing, designing scalable, resilient and maintainable data solutions.
• Proven ability to analyse complex requirements, produce clear specifications and define effective testing approaches.
• Experience leading the resolution of complex data, platform and pipeline issues across strategic, tactical and operational levels.
• Strong knowledge of migrating and transforming data from legacy and on-premise systems to cloud platforms.
• Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments.

You and your role

As a Data Engineer, you will design build and enhance scalable ETL/ELT pipelines using Python and PySpark, ensuring data solutions are reliable, efficient and aligned with engineering best practice.

You’ll work closely with stakeholders to understand complex business requirements, transforming data into trusted, high-quality datasets that enable valuable insights, reporting and decision-making.

You will manage and optimise data across multiple platforms, ensuring it is secure, accessible and fit for purpose, while supporting effective database management and data extraction solutions.

Leading teams through large-scale data and platform transformation, you’ll help shape technical direction, embed engineering best practice and develop capability across the wider team.

Using your cloud data engineering expertise, you’ll develop complex data pipelines, drive automation, improve platform performance and champion Agile delivery, data governance and operational excellence to deliver resilient, high-quality data services.

Location:

You will join us in your choice of Birmingham, Blackpool, Coventry, Darlington, Leeds, London (Caxton House), Manchester, Newcastle Upon Tyne and Sheffield

Hybrid Working:

We work a hybrid model - you’ll spend some time working at home and some time collaborating face to face in a hub.

Pay:

We offer competitive pay of up to £75,664. You’ll get a salary of up to £68,205, and a Digital Allowance of up to £7,459 is available for exceptional candidates – both elements of the offer are subject to our assessment of your capability at interview.

Pension:

You’ll get a brilliant civil service pension with employer contributions worth 28.97%, worth up to £16,786 per year.

Holidays:

A generous leave package starting at 26 days rising to 31 days over time.

You can also take up to 3 extra days off a month on flexi-time. You’ll also get all the usual public holidays.

We have a broad benefits package built around your work-life balance which includes:

• Flexible working including flexible hours and flex-friendly policies
• Time off volunteering and charitable giving
• Bring your authentic self to work with 'I Can Be Me in DWP'
• Discounts and savings on shopping, fun days out and more
• Interest-free loans to buy a bike or a season ticket, so it’s even easier for you to get to work and start making a difference
• Sports and social activities
• Professional development, coaching, mentoring and career progression opportunities.

And we have an award-winning environment and culture:

• DWP have been recognised as 2024 Diversity Employer of the Year at the Computing Women in Tech Excellence awards
• Diverse and Inclusive Leadership at Digital Leaders Awards 2024
• Commended as Best Place to Work in Digital category in the Computing Digital Technology Leaders awards 2025
• Recognised as one of the Best Public Sector Employers at 2025 Women In Tech Employer Awards