Senior Data Engineer

Allscreens Nationwide Ltd; Newcastle upon Tyne; On-site; GBP 43,000 - 50,000; Full time

Benefits offered by this job

Disability Confident employer scheme

Job summary

Government Recruitment Service in the United Kingdom is seeking a Senior Data Engineer to drive enterprise data engineering initiatives within the Defra data ecosystem. You will lead cloud-based data platforms, design robust ETL pipelines, and collaborate with cross-government teams to unlock data value for analytics and decision-making.

The role emphasizes leadership, mentoring, and governance across data assets, with a permanent, full-time arrangement based in Newcastle-upon-Tyne.

Qualifications

• Experience designing cloud-based data platforms and architectures.
• Experience building and optimising ETL pipelines for analytics.
• Knowledge of data governance, metadata management, and data lineage.
• Ability to collaborate with Defra stakeholders and mentor peers.

Responsibilities

• Design, build, and maintain cloud-based data platforms and architectures that integrate large, complex datasets, ensuring scalability, security, and performance.
• Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use.
• Implement robust data models and storage solutions that support efficient querying, data integrity, and enterprise-wide accessibility.
• Collaborate with stakeholders across Defra group to understand data requirements, integrate multiple sources, and deliver solutions that meet business needs.
• Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking to maintain trusted datasets.
• Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement within the team.
• Lead technical workstreams and mentor colleagues, fostering a culture of collaboration, knowledge sharing, and engineering excellence.

Skills

Leadership

Mentoring

Data governance

Data modelling

Communication

Tools

Azure Synapse

Databricks

Python

SQL

ETL pipelines

Job description

The Data and Analysis for Delivery (DA4D) division, part of Delivery and Group Corporate Services in Defra, plays a pivotal role in enabling data-driven decision-making across the department’s major delivery portfolio. Within DA4D, the Delivery Data team is responsible for the collection, management, and exploitation of data relating to the major delivery portfolio, performance and risk. By ensuring accurate, timely, and accessible data, the team supports Defra in understanding delivery progress and making informed strategic decisions. Its work underpins the department’s ability to manage complex programmes effectively and achieve outcomes that benefit the departments core purpose.

You will play a key role in driving improvements in how Defra collects, manages, and harnesses data at enterprise level.

Working closely with delivery teams, you will help identify, manage, and unlock the value of critical data assets, enabling evidence-based decision-making and more efficient operations.

Your day-to-day work will be varied and impactful. One day you may be designing and scoping robust data architectures or modern storage solutions to enhance data quality, integrity, and accessibility across the organisation. On another, you might be developing and optimising data pipelines that deliver trusted, high-quality data to data scientists, analysts, and self-service users, supporting their need for timely and accurate insights.

Collaboration will be at the heart of your role. You will work with a wide range of colleagues from across Defra, its arm’s length bodies, and external partners, optimising data flows, integrating with multiple data collection platforms, and ensuring data solutions are secure, scalable, and fit for purpose. Through your work, you will help shape and deliver Defra’s ambition to become a data-driven organisation.

The team has recently delivered a critical milestone: an end-to-end Delivery Data Service, including a platform for collecting data, pipelines for ingesting and cleaning data, and a data science hub for analysis, visualisation, and interpretation. This foundation enables key services such as Portfolio Management, Enterprise Performance, and Enterprise Risk.

Building on this, you will lead data engineering workstreams that enhance and scale our data architecture, optimise data flows, and deliver new pipelines to bring critical datasets into the team. Your focus will be on designing and implementing robust, cloud-based solutions that ensure data is accurate, accessible, and performant across the organisation. You’ll work with technologies such as Azure Synapse, Databricks, Python, and SQL to develop ETL processes, improve data quality, and enable downstream analytics and reporting.

Collaboration will be central to your role. You’ll partner with colleagues across Defra group to integrate multiple data sources, streamline ingestion processes, and ensure solutions are secure, scalable, and aligned with best practices. This is your opportunity to shape how Defra manages and exploits data at enterprise scale, supporting one of the most complex delivery portfolios in government.

You’ll thrive in a fast-paced, evolving environment - seeing opportunity in ambiguity, adapting quickly, and learning openly. As part of a small, ambitious, and friendly team, you’ll combine technical expertise with creativity to deliver meaningful change.

Key Responsibilities:

• Design, build, and maintain cloud-based data platforms and architectures that integrate large, complex, and disparate datasets, ensuring scalability, security, and performance.
• Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use.
• Implement robust data models and storage solutions that support efficient querying, data integrity, and enterprise-wide accessibility.
• Collaborate with stakeholders across Defra group to understand data requirements, integrate multiple sources, and deliver solutions that meet business needs.
• Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking to maintain trusted datasets.
• Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement within the team.
• Lead technical workstreams and mentor colleagues, fostering a culture of collaboration, knowledge sharing, and engineering excellence.

Proud member of the Disability Confident employer scheme

Senior Data Engineer

Company: Government Recruitment Service

Salary: £42,665 - £50,495

Hours: Full-time

Location: Newcastle-upon-Tyne

Job type: Permanent

Posting date: 7 Aug 2026

Closing date: 21 Aug 2026

About Disability Confident

Disability Confident is a government scheme. It encourages employers to recruit and retain disabled people and those with long term health conditions.