Lead Data Engineer

UK Home Office; Sheffield; Híbrido; GBP 56.000 - 69.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Civil Service Pension

25 days annual leave

8 public holidays

Hybrid work 60% in office

Descripción de la vacante

The UK Home Office is seeking a Lead Data Engineer for the National Data Quality Improvement Service (NDQIS) to design, deliver, and support end-to-end data solutions. You will translate business requirements into robust pipelines, ensuring governance, security, and reliability across data integrations and automated workflows.

You will drive the data engineering vision, promote best practices, and lead a team to build scalable data products while collaborating with analysts and stakeholders to

Formación

• Design, deliver and support end-to-end data solutions for data-driven decision making.
• Translate business requirements into robust, scalable data pipelines with governance and security in mind.
• Lead data integrations, ETL processes and automated workflows from multiple sources.

Responsabilidades

• Manage data sources to improve quality and preparation for analysis.
• Develop and support complex data workflows and large-scale data products across the delivery lifecycle.
• Lead stakeholder engagement and manage expectations while fostering a data-focused service culture.
• Ensure data privacy and information security standards are met in solutions.

Conocimientos

Python

NLP

API integration

SQL

Data analysis

Herramientas

AWS

Docker

Git

Github

CI/CD pipelines

Python-based automation

Descripción del empleo

Location: hybrid with 60% attendance at either Salford | Croydon | Sheffield

Salary: £62,732 plus skills allowance of up to £17,668

Advert Close Date: Monday 9th November 11:55pm

Job summary

This role will be based in National Data Quality Improvement Service (NDQIS) team. NDQIS is a national service for policing that uses rule based to improve the quality, consistency and comparability of police recorded crime data submitted to the Home Office.

As a Lead Data Engineer, you will be responsible for designing, delivering and supporting high-quality end-to-end data solutions that enable effective data-driven decision making. Working across the full delivery lifecycle, you will translate complex business requirements into robust, scalable and secure data pipelines, ensuring compliance with Home Office data quality, governance and security standards.

You will lead the design and implementation of complex data integrations and text analysis methodologies, ETL processes and automated workflows, bringing together information from multiple internal and external sources. Working closely with technical and business stakeholders, you will ensure solutions are reliable, resilient and aligned with organisational objectives.

You will shape and implement the data engineering vision and roadmap, promoting best practice, innovation and continuous improvement. Driving automation and championing data quality, you will ensure solutions remain maintainable, cost-effective and scalable. You will build strong relationships across technical and non-technical communities, influencing decision-making and ensuring services align with organisational priorities and user needs.

Main Responsibilities

• Managing data sources to improve data quality and conform them for analysis, developing processes to balance data availability, capacity and data archiving, while continually improving processes and efficiency in the acquisition and ingestion of data.
• Developing, maintaining and supporting data workflows, pipelines, ETL processes and end-to-end data solutions covering requirements review, data profiling and source system analysis, design, coding, testing, deployment and documentation of complex and large-scale data products, while working collaboratively with the analytics team to ensure the reliability, usability and integrity of critical datasets.
• Exercising key stakeholder management skills in order to manage expectations, listen and interpret business needs, communicate progress and facilitate difficult conversations, while building a 'service culture' around data.
• Keeping abreast of best practice in industry and government, ensuring data solutions comply with data privacy and information security requirements, working closely with relevant specialists.
• Managing the performance of suppliers and the data engineering team to improve capability, as well as creating the roadmap which balances immediate user needs with long-term investment, evaluation and selection of software products to build data engineering capability, contributing to the re-platforming of data analytics systems and the refresh of tooling.
• Line management, coaching and professional development of the team, where required.
• Effectively leading and managing teams, leading and delivering technical products and projects in data analysis, solution design and end user reporting.
• Translating natural-language data quality rules into automated, testable and documented pipeline logic, applying modern programming languages and data analysis techniques (particularly Python and related libraries, regex and NLP), alongside qualitative and quantitative analysis, API integration, and JSON/CSV manipulation.
• Advising and leading the use of best practice IT programming and automation techniques and tools e.g. Git, Github, CI/CD pipelines & Python based automation.
• Leading the use of Cloud Data technologies, solutions and future Cloud Data Strategies g. AWS, EC2, VPC, RDS, IAM, R53, Lambda, Cloudwatch, AD & Docker.
• Working with stakeholders to gather requirements and deliver findings, presenting complex ideas and recommendations that inform decisions in a clear way to technical and non-technical audiences and senior customers.
• Using tools such as industry standard ETL tools, network databases and scheduling and orchestration tools g. SQL, Linux, bash scripts, PostgreSQL, Apache Derby, AWS Glue & Step Functions.
• Familiarity with data visualisation tools (e.g. Power BI), and emerging tools such as LLMs.
• Data engineering in the context of AI, Machine Learning, or automation-enabled products or services.

Why work for us...

Find out more information at: Benefits - Home Office Careers, but some of the primary ones are:

• A Civil Service Pension with employer contribution rates of at least 28.97%.
• In-year reward scheme for one-off or sustained exceptional personal or team achievements.
• 25 days annual leave on appointment, rising with service.
• 8 days of public holidays, plus 1 additional privilege day.
• Where business needs allow, some roles may be suitable for a combination of office and home-based working. This is a non-contractual arrangement where all employees will be expected to spend a minimum of 60% of their working time in an office.