Lead Data Engineer

HM TREASURY-1; North East; On-site; GBP 50,000 - 80,000; Full time

Job summary

A government department in the UK is seeking a Lead Data Engineer to develop and maintain AI products and collaborate on data strategies. Candidates should have strong Python expertise and experience with cloud platforms like Azure. The role offers flexible working patterns, generous leave, and a competitive pension scheme.

Benefits

25 days annual leave, rising to 30 after 5 years

Flexible working patterns

Generous parental and adoption leave packages

Defined Benefit pension scheme with 28.97% employer contributions

Qualifications

• Strong experience using Python for data engineering and web app development.
• Experience with at least one major cloud provider.
• Understanding of LLM and embedding APIs.

Responsibilities

• Acquire data sets from multiple sources of varying maturity.
• Deliver streaming services by linking data from multiple systems.
• Develop resilient, scalable, and future-proof data services.
• Promote best practices in data engineering.

Skills

Python

SQL

Cloud systems

Data engineering tools

AI knowledge (LLMs and embeddings)

Tools

Azure

Google Cloud

AWS

Job description

Overview

Are you an experienced and successful Data Engineer looking for a new, challenging and rewarding career? If so, we have the perfect job for you !

About the Team

The Data Hub is a centre of excellence for data management and advanced analytics in HM Treasury. Working closely partners inside and outside of government, the Hub is responsible for the implementation of the Treasury's data strategy, developing the organisation's data capability and driving data-led transformation across the department. The Hub also provides strategic coordination and support to teams across the department, bringing data tools and technologies to bear on new and existing data sources. It apples innovative methods to enable rich analysis and support HMT's departmental objectives, delivering new insights and improving decision-making. To do this, the Hub is organised into two functions:

• Data Management - leads the department in developing its data management architecture, engineering, and data literacy. This includes developing systematic and structural approaches using data science techniques and ensuring that officials at all levels have access to data-related learning and development opportunities.
• Advanced Analytics - a flexible pool of data scientists who are responsible for leading the development of the department's AI capabilities, and can identify, initiate, and support high-value data science projects across the department.

About the Job

In this role:

• You will be acquiring data sets from multiple sources at varying levels of maturity.
• You will have a demonstrable understanding of how to expose data from systems (e.g. from APIs), link data from multiple systems, and deliver streaming services.
• You will collaborate with others to identify and plan these services, and to review specifications.
• You will maintain and develop AI products used internally and develop new products in line with HMT's AI roadmap.
• You will leverage your expertise in Python, cloud systems and AI (specifically LLMs and embeddings) to achieve this.
• You will ensure the products run reliably and react to any issues rapidly.
• You can produce data models and understand where to use different types of data models.
• You have experience of a wide range of data engineering tools.
• You understand industry-recognised data-modelling patterns, how to establish and use agreed standards and tools, design, code, and test programs and scripts for data engineering.
• You can develop resilient, scalable, and future-proof services for to meet user needs.
• You will actively scope and carry out other projects including web app development and data engineering.
• You will explore opportunities to work innovatively, recognising, identifying and exploiting opportunities to ensure efficient and effective performance of the organisation, and to support and enable that you will influence the data architecture in place.
• You can explore new ways of conducting business and organisational processes.
• You will promote best practice in data engineering including acquisition, storage and sharing of data for the benefit of modelling and policy development, working with the Hub to provide overall technical leadership and development opportunities across HMT in data and data science.

As a lead data engineer, you will work on a variety of data engineering projects and own several AI products, maintaining and developing them in response to user feedback. These AI products are used by most HMT staff: improving them will increase productivity across the department.

About You

You will have strong experience using Python for data engineering and web app development; expertise in SQL ; considerable experience at least one major Cloud provider (Azure is preferred, however Google Cloud or AWS are acceptable); experience successfully implementing a product and managing customer relationships, acting on feedback and other good product management practices and experience using LLM and Embedding APIs, with a strong understanding of how both work.

Some of the Benefits our people love!

• 25 days annual leave (rising to 30 after 5 years), plus 8 public holidays and the King\'s birthday (unless you have a legacy arrangement as an existing Civil Servant). Additionally, we operate flexitime systems, allowing employees to take up to an additional 2 days off each month
• Flexible working patterns (part-time, job-share, condensed hours)
• Generous parental and adoption leave packages
• Access to a generous Defined Benefit pension scheme with employer contributions of 28.97%
• Access to a cycle-to-work salary sacrifice scheme and season ticket advances
• A range of active staff networks, based around interests (e.g. analysts, music society, sports and social club) and diversity

For more information about the role and how to apply, please follow the apply link.

If you need any reasonable adjustments to take part in the selection process, please tell us about this in your online application form, or speak to the recruitment team at .