Senior Data Engineer (Azure) - Lead Data Pipelines, Hybrid

DWP Digital; Coventry; Hybrid; GBP 45.000 - 76.000; Vollzeit

Benefits dieser Stelle

Flexible hours

Hybrid working

Professional development and mentoring

Pension contributions (28.97%)

Discounts and savings

Zusammenfassung

DWP Digital in the United Kingdom is seeking a Senior Data Engineer to design, build and maintain cloud-based data pipelines using Python and PySpark. You will help ensure trusted data for decision-making and work with stakeholders across the organisation.

You will lead data engineering initiatives, mentor teammates and drive automation and governance while embracing Agile delivery and hybrid working arrangements.

Qualifikationen

• Proven experience designing and delivering complex ETL/ELT pipelines.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences, including senior stakeholders.
• Strong understanding of cloud and distributed data processing, designing scalable, resilient and maintainable data solutions.
• Proven ability to analyse complex requirements, produce clear specifications and define effective testing approaches.
• Experience leading the resolution of complex data, platform and pipeline issues across strategic, tactical and operational levels.
• Strong knowledge of migrating and transforming data from legacy and on-premise systems to cloud platforms.
• Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments.

Aufgaben

• Design, build and enhance scalable ETL/ELT pipelines using Python and PySpark, ensuring data solutions are reliable and aligned with engineering best practice.
• Work closely with stakeholders to understand complex business requirements and transform data into trusted datasets for insights.
• Manage and optimise data across multiple platforms, ensuring security, accessibility and fit for purpose.
• Lead teams through large-scale data and platform transformation and develop capability across the team.
• Develop complex data pipelines, drive automation and champion Agile delivery, data governance and operational excellence.

Kenntnisse

ETL/ELT pipeline design

PySpark

Spark SQL

Python

Cloud data engineering

Stakeholder communication

Leadership

Agile delivery

Tools

Azure

Spark

Jobbeschreibung

DWP Digital in the United Kingdom is seeking a Senior Data Engineer to design, build and maintain cloud-based data pipelines using Python and PySpark. You will help ensure trusted data for decision-making and work with stakeholders across the organisation.

You will lead data engineering initiatives, mentor teammates and drive automation and governance while embracing Agile delivery and hybrid working arrangements.