Senior Data Engineer

Allscreens Nationwide Ltd; Greater London; On-site; GBP 43,000 - 50,000; Full time

Benefits offered by this job

Disability Confident employer scheme

Job summary

Government Recruitment Service is seeking a Senior Data Engineer in London to drive data architecture improvements and enterprise-level data solutions. You will design cloud data platforms, build scalable pipelines, and collaborate with Defra teams to unlock data value for decision-making.

The role emphasises data quality, governance, and modern technologies such as Azure Synapse, Databricks, Python, and SQL.

Qualifications

• Design robust data architectures and cloud-based storage solutions.
• Develop and optimise ETL pipelines for analytics and reporting.
• Collaborate with Defra stakeholders to integrate data sources and deliver scalable solutions.

Responsibilities

• Design, build, and maintain cloud-based data platforms and architectures.
• Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use.
• Implement robust data models and storage solutions that support efficient querying and enterprise-wide accessibility.
• Collaborate with stakeholders across Defra group to understand data requirements and deliver solutions that meet needs.
• Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking.
• Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement.
• Lead technical workstreams and mentor colleagues, fostering collaboration and engineering excellence.

Skills

Azure Synapse

Databricks

Python

SQL

ETL

Data pipelines

Job description

Senior Data Engineer

Company: Government Recruitment Service

Salary: £42,665 - £50,495

Hours: Full-time

Location: London

Job type: Permanent

Posting date: 7 Aug 2026

Closing date: 21 Aug 2026

The Data and Analysis for Delivery (DA4D) division, part of Delivery and Group Corporate Services in Defra, plays a pivotal role in enabling data-driven decision-making across the department’s major delivery portfolio. Within DA4D, the Delivery Data team is responsible for the collection, management, and exploitation of data relating to the major delivery portfolio, performance and risk. By ensuring accurate, timely, and accessible data, the team supports Defra in understanding delivery progress and making informed strategic decisions. Its work underpins the department’s ability to manage complex programmes effectively and achieve outcomes that benefit the departments core purpose.

You will play a key role in driving improvements in how Defra collects, manages, and harnesses data at enterprise level.

Working closely with delivery teams, you will help identify, manage, and unlock the value of critical data assets, enabling evidence-based decision-making and more efficient operations.

Your day-to-day work will be varied and impactful. One day you may be designing and scoping robust data architectures or modern storage solutions to enhance data quality, integrity, and accessibility across the organisation. On another, you might be developing and optimising data pipelines that deliver trusted, high-quality data to data scientists, analysts, and self-service users, supporting their need for timely and accurate insights.

Collaboration will be at the heart of your role. You will work with a wide range of colleagues from across Defra, its arm’s length bodies, and external partners, optimising data flows, integrating with multiple data collection platforms, and ensuring data solutions are secure, scalable, and fit for purpose. Through your work, you will help shape and deliver Defra’s ambition to become a data-driven organisation.

The team has recently delivered a critical milestone: an end-to-end Delivery Data Service, including a platform for collecting data, pipelines for ingesting and cleaning data, and a data science hub for analysis, visualisation, and interpretation. This foundation enables key services such as Portfolio Management, Enterprise Performance, and Enterprise Risk.

Building on this, you will lead data engineering workstreams that enhance and scale our data architecture, optimise data flows, and deliver new pipelines to bring critical datasets into the team. Your focus will be on designing and implementing robust, cloud-based solutions that ensure data is accurate, accessible, and performant across the organisation. You’ll work with technologies such as Azure Synapse, Databricks, Python, and SQL to develop ETL processes, improve data quality, and enable downstream analytics and reporting.

Collaboration will be central to your role. You’ll partner with colleagues across Defra group to integrate multiple data sources, streamline ingestion processes, and ensure solutions are secure, scalable, and aligned with best practices. This is your opportunity to shape how Defra manages and exploits data at enterprise scale, supporting one of the most complex delivery portfolios in government.

You’ll thrive in a fast-paced, evolving environment - seeing opportunity in ambiguity, adapting quickly, and learning openly. As part of a small, ambitious, and friendly team, you’ll combine technical expertise with creativity to deliver meaningful change.

Key Responsibilities:

• Design, build, and maintain cloud-based data platforms and architectures that integrate large, complex, and disparate datasets, ensuring scalability, security, and performance.
• Develop and optimise ETL pipelines to ingest, transform, and deliver high-quality data for analytics, reporting, and operational use.
• Implement robust data models and storage solutions that support efficient querying, data integrity, and enterprise-wide accessibility.
• Collaborate with stakeholders across Defra group to understand data requirements, integrate multiple sources, and deliver solutions that meet business needs.
• Ensure data quality and governance by applying validation frameworks, metadata management, and lineage tracking to maintain trusted datasets.
• Stay current with emerging data engineering tools and best practices, contributing to innovation and continuous improvement within the team.
• Lead technical workstreams and mentor colleagues, fostering a culture of collaboration, knowledge sharing, and engineering excellence.

Proud member of the Disability Confident employer scheme

About Disability Confident Disability Confident is a government scheme. It encourages employers to recruit and retain disabled people and those with long term health conditions.

Related jobs

Senior Databricks Engineer

£300 to £325 per day

SK IT SOLUTIONS LTD

1000 Great West Road

Hybrid Contract Full time

Junior Databricks Engineer

£225 to £250 per day

SK IT SOLUTIONS LTD

1000 Great West Road

Hybrid Contract Full time

Principal Software Engineer AI Native

Aventum Group

London

Hybrid Permanent Full time

Browse more jobs