Senior Data Engineer: Azure Data Pipelines Lead

DWP; Lancashire; Híbrido; GBP 45 000 - 76 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Hybrid working

Flexible hours

Civil service pension with 28.97%贡献

Resumo da oferta

DWP is seeking a Data Engineer to design, build and enhance scalable ETL/ELT pipelines using Python and PySpark, ensuring data solutions are reliable and efficient. The role requires leadership of technical teams to deliver cloud-based data platforms that support decision-making across the organisation.

Successful candidates will work across multiple data domains, collaborating with stakeholders to transform requirements into trusted data assets while adhering to engineering standards and

Qualificações

• Proven experience designing and delivering complex ETL/ELT pipelines, managing priorities and driving quality outcomes.
• Ability to communicate complex technical concepts clearly to technical and non-technical audiences, including senior stakeholders.
• Strong understanding of cloud and distributed data processing, designing scalable, resilient and maintainable data solutions.
• Proven ability to analyse complex requirements, produce clear specifications and define effective testing approaches.
• Experience leading the resolution of complex data, platform and pipeline issues across strategic, tactical and operational levels.
• Strong knowledge of migrating and transforming data from legacy and on-premise systems to cloud platforms.
• Expertise in automating ETL/ELT pipelines using PySpark and Spark SQL across Azure-based data environments.

Responsabilidades

• design, build and enhance scalable ETL/ELT pipelines using Python and PySpark, ensuring data solutions are reliable, efficient and aligned with engineering best practice.
• transform data into trusted, high-quality datasets that enable valuable insights, reporting and decision-making.
• manage and optimise data across multiple platforms, ensuring it is secure, accessible and fit for purpose, while supporting effective database management and data extraction solutions.
• Lead teams through large-scale data and platform transformation, help shape technical direction, embed engineering best practice and develop capability across the wider team.
• Using cloud data engineering expertise, develop complex data pipelines, drive automation, improve platform performance and champion Agile delivery, data governance and operational excellence.

Conhecimentos

ETL/ELT pipelines

Azure

Data pipelines

Leadership

Communication

Cloud computing

PySpark

Spark SQL

Ferramentas

PySpark

Spark SQL

Azure

Descrição da oferta de emprego

DWP is seeking a Data Engineer to design, build and enhance scalable ETL/ELT pipelines using Python and PySpark, ensuring data solutions are reliable and efficient. The role requires leadership of technical teams to deliver cloud-based data platforms that support decision-making across the organisation.

Successful candidates will work across multiple data domains, collaborating with stakeholders to transform requirements into trusted data assets while adhering to engineering standards and