Lead Data Engineer

LGBT Jobs; West of England; Hybrid; GBP 90,000 - 120,000; Full time

Job summary

National Highways is seeking a Lead Data Engineer to drive the Data as a Service roadmap and deliver trusted data products for the transport network. The role focuses on designing and enhancing data pipelines, governing data and enabling decision-making across the organisation.

The successful candidate will collaborate with stakeholders, implement enterprise data capabilities, and promote best practices in data engineering.

Qualifications

• Experience building data pipelines in a collaborative delivery environment with governance practices.
• Experience analysing source systems and complex datasets to support analytics and reporting.
• Proficiency with Python and SQL against enterprise data platforms and cloud ecosystems (Azure Databricks, Unity Catalog preferred).
• Strong communication with technical and non-technical stakeholders, owning delivery outcomes.
• Experience operating and improving production data pipelines and data products, including monitoring and cost optimization.

Responsibilities

• Deliver the Data as a Service (DaaS) roadmap by onboarding enterprise data and expanding analytical capabilities.
• Design, build and enhance data pipelines, integrations and reusable data assets across transactional, asset, traffic, event and business data.
• Collaborate with stakeholders to understand analytical needs, develop user stories and deliver trusted data products for operational and strategic decisions.
• Embed governance, quality and assurance across the data lifecycle, ensuring policy compliance and metadata management (Unity Catalog).
• Support operation and continuous improvement of production data pipelines and platform capabilities for reliability and cost efficiency.
• Promote best practices and reusable engineering approaches across the organisation.

Skills

Data pipelines

Data governance

Python

SQL

Cloud platforms

Tools

Databricks

Power BI

ThoughtSpot

Unity Catalog

Azure Data Factory

Azure Cloud

Job description

Lead Data Engineer role pivotal for developing digital solutions in transport network

Hybrid role requiring two office days per week at any National Highways office location

Responsibilities include designing and enhancing data pipelines, and ensuring data governance

Requires experience with enterprise data platforms and cloud-based ecosystems

Role promotes data engineering best practices and collaborative decision-making

About the role.

As part of the development of our Digital Services function, we are strengthening internal digital capability within the Data Team. These roles are central to building sustainable in‑house expertise that supports the design, development, and delivery of digital solutions across the transport network.

A key focus of this campaign is creating new opportunities, and as a result, we are recruiting Lead Data Engineers. This is a pivotal role within the organisation.

In this role, you will work as part of a collaborative team, partnering with stakeholders across the business to understand user needs and deliver data products that create real organisational value. Alongside developing new capabilities, you will help ensure our existing data services remain reliable, performant and fit for purpose.

This is an opportunity to shape how data is used across one of the UK's largest and most complex transport organisations, contributing to safer, smoother and more reliable journeys for road users.

This is a hybrid role, requiring two days per week in the office, based at any National Highways office location.

• Deliver the Data as a Service (DaaS) roadmap by onboarding enterprise data, improving trust in data and expanding the organisation's analytical and reporting capabilities.
• Design, build and enhance data pipelines, integrations and reusable data assets that bring together transactional, asset, traffic, event and business data within the DaaS platform.
• Collaborate closely with stakeholders at all levels across the organisation to understand analytical and reporting needs, develop user stories and deliver trusted data products that support operational and strategic decision-making. Enable consumers through enterprise solutions such as: Databricks, Power BI and ThoughtSpot.
• Embed data governance, quality and assurance across the full data lifecycle, ensuring compliance with organisational policy, responsible use of AI and metadata management. Governance is enabled through Unity Catalog.
• Support the operation and continuous improvement of production data pipelines, data products and platform capabilities, improving reliability, performance, cost efficiency and user experience.
• Promote data engineering best practice, reuse and consistent engineering approaches, helping teams make effective use of enterprise data capabilities across the organisation.

About you.

• Experience building and deploying data pipelines and data integration solutions within a collaborative delivery environment, applying engineering standards, automation, testing and governance practices throughout the development lifecycle.
• Experience analysing source systems and complex datasets, bringing together data from multiple sources and applying data modelling or related design techniques to create reusable data assets that support analytics, reporting and business outcomes.
• Experience working with scripting Command-Line Interface (CLI), Python and SQL with enterprise data platforms and cloud-based data ecosystems (Azure Cloud, Databricks, Azure Data Factory, Unity Catalog preferred).
• Experience communicating effectively with technical and non-technical stakeholders, managing expectations, taking ownership of delivery outcomes and influencing decisions, improvements or ways of working through technical expertise and collaboration.
• Experience operating and improving production data pipelines and data products, including monitoring, diagnosing and resolving incidents, root cause analysis, performance and cost optimisation and service reliability.

About us.

Here at National Highways, we manage and improve England's motorways and major A roads, helping our customers have safer, smoother and more reliable journeys. Our priorities are safety, customers and delivery, and at the core of this, are our values of passion, integrity, safety, teamwork and ownership.

The Digital Services Directorate is helping to reshape the way National Highways operates by delivering digital, data, and technology services across all areas, from frontline to back office, in a modern and efficient manner. Our vision is to develop integrated information and technology that empowers our colleagues and provides real-time information to our customers, as well as integrating with intelligent vehicle and transport systems as they evolve, to enhance journey safety and reliability.

External candidates will be offered a starting salary at the lower end of the pay scale, while current employees will be appointed in accordance with our established pay policy.

We are committed to creating a diverse environment and welcome applicants from all backgrounds.

If this job opportunity is not suitable, then check out the other LGBT friendly jobs across UK!