Data Engineer

Government Recruitment Service; York and North Yorkshire; On-site; GBP 50,000 - 70,000; Full time

Job summary

A governmental agency is seeking a Data Engineer to enhance their Data Analytics Service Team. The role involves reviewing and streamlining ETL processes, ensuring data quality in pipelines, and leading initiatives to optimize Azure resources. The ideal candidate will collaborate with civil service partners and adapt data handling practices in line with evolving best practices. Strong knowledge of Azure and experience in data processing are essential for this position.

Responsibilities

• Review, design, document, and streamline ETL processes for datasets.
• Ensure new datasets are added with correct processes.
• Propose changes based on best practices in data management.

Skills

ETL processes

Data processing in Azure

Leadership

Collaboration with data providers

Tools

Azure

Databricks

Job description

This role sits within the Data Analytics Service Team. You will be working as our Data Engineer and will be required to review/ design, document, and streamline the ETL processes for the datasets we bring together for analysis and AI projects. We operate in Azure using ABFSS for storage and Databricks Unity catalogue for our data processing. As part of the Senior Management Team, you will ensure new datasets are being added with the correct processes, existing pipelines execute correctly, and proposing changes to these processes going forward in line with best practices.

As part of the senior management of the data analytics team, you will also have leadership responsibilities within the PSFA data team to help us improve our services.

You will work with the platform lead to identify where and how we should continue to build our Azure tenant to best support the ambitions of the PSFA data analytics service going forward. You will develop relationships with key partners across the civil service, including our key data providers and our cloud provider, to identify how we can work together to improve our tenant.