Lead Data Engineer

Manchester Digital; Salford; On-site; GBP 60,000 - 80,000; Full time

Job summary

A leading data services provider in the UK is seeking two Lead Data Engineers to develop and manage cloud-based data solutions. The roles heavily revolve around AWS data pipelines, DevOps automation, and stakeholder engagement. Candidates should be experienced in Python, SQL, and have solid data quality management skills. This position requires UK residency for security clearance and offers the opportunity to manage a team and shape the data strategy across various digital services.

Qualifications

• Experience developing scalable data engineering solutions.
• Proficient in managing data models and ETL processes.
• Strong communication skills for stakeholder management.

Responsibilities

• Manage data sources to improve quality and readiness for analysis.
• Develop new data models and ETL processes in collaboration with analytics.
• Ensure supplier performance and manage the data team.
• Continuously improve data acquisition and ingestion processes.
• Line manage staff with a focus on coaching and development.

Skills

AWS data pipelines

DevOps automation

Terraform

Java

Python

SQL

PowerShell

Data quality management

Stakeholder management

Tools

AWS

Azure

Kafka

Glue

Fabric

Job description

Overview

Lead Data Engineer x2 (DSA) - Data Services and Analytics (DSA) is the part of HO Digital which leads the build of data products to integrate data from multiple sources, and then deliver products for business intelligence, reporting and data science. DSA has the ambition to be one of the leading providers of insight services in UK Government and is currently growing significantly with over 300 colleagues, providing services to variety of functions within the Home Office.

These roles focus on delivering scalable cloud based data engineering by developing complex AWS data pipelines and high quality data models. With a large focus being DevOps automation with Terraform or AWS CloudFormation, Java based service support, Dynatrace monitoring integration, and managing CI/CD pipelines using Drone. You’ll also drive event driven architectures with Kafka and ensure reliable, well governed datasets that enable data driven decision making across the organisation.

Lead Data Engineer x 2 (EUC&C) – The End User Compute and Collaboration (EUC&C) team develops and delivers a range of Microsoft 365 solutions, including Teams, SharePoint, OneDrive, Power Platform, and Office applications. These tools support collaboration and productivity across the organisation. We are establishing a new data capability to drive the adoption of data driven insights across the organisation’s product and service landscape. Working on end-to-end solutions in both AWS and Azure, using languages including Python, SQL, and PowerShell and orchestration platforms such as Glue and Fabric, you will lead the design, development, and maintenance of robust, scalable data pipelines. You’ll work closely with senior data colleagues and key stakeholders to shape and deliver EUC&C’s data vision.

Responsibilities

Your main day to day responsibilities will be:

• Managing data sources in a way to improve data quality and conform them for analysis.
• Developing new data models and ETL processes, working collaboratively with the analytics team and mapping data to an appropriate matching data model.
• Exercising key stakeholder management skills in order to manage expectations, listen and interpret business needs, communicate progress and facilitate difficult conversations, while building a \"service culture\" around data.
• Managing the performance of suppliers and the data team to improve capability, as well as creating the roadmap which balances immediate user needs with long-term investment.
• Developing processes to balance data availability, capacity and data archiving, while continually improving processes and efficiency in the acquisition and ingestion of data.
• Line managing staff, including coaching and professional development.

UK residency and security requirements

Please note that this role requires SecurityCheck(SC) clearance, which would normally need 5 years’ UK residency in the past 5 years.

However, in exceptional circumstances security clearance applications for candidates who have been present in the UK for at least 3 of the last 5 years may be considered. Failure to meet this residency requirement will result in your security clearance application being rejected.