Data Engineer

Government Recruitment Service; Glasgow; On-site; GBP 40,000 - 60,000; Full time

Job summary

A government agency in the UK is seeking a skilled Data Engineer to streamline ETL processes and enhance data services. The Data Engineer will work within the Data Analytics Service Team, ensuring new datasets are integrated effectively while adhering to best practices. Leadership responsibilities will also be essential, improving teamwork and collaboration with civil service partners. Ideal candidates will have experience in Azure and strong data engineering skills, making a significant impact on the agency’s data ambitions.

Qualifications

• Experience in managing ETL processes for data analysis.
• Proficiency in Azure services and data management.
• Strong leadership skills within data teams.

Responsibilities

• Review, design, document, and streamline ETL processes.
• Ensure new datasets are added with correct processes.
• Lead the PSFA data team to improve services.

Skills

Data Engineering

ETL processes

Azure

Databricks

Leadership

Job description

This role sits within the Data Analytics Service Team. You will be working as our Data Engineer and will be required to review/ design, document, and streamline the ETL processes for the datasets we bring together for analysis and AI projects. We operate in Azure using ABFSS for storage and Databricks Unity catalogue for our data processing. As part of the Senior Management Team, you will ensure new datasets are being added with the correct processes, existing pipelines execute correctly, and proposing changes to these processes going forward in line with best practices.

As part of the senior management of the data analytics team, you will also have leadership responsibilities within the PSFA data team to help us improve our services.

You will work with the platform lead to identify where and how we should continue to build our Azure tenant to best support the ambitions of the PSFA data analytics service going forward. You will develop relationships with key partners across the civil service, including our key data providers and our cloud provider, to identify how we can work together to improve our tenant.