Senior Data Engineer

The Alan Turing Institute; City Of London; On-site; GBP 57,000 - 69,000; Full time

Benefits offered by this job

Flexible working

Family-friendly policies

Job summary

The Alan Turing Institute is seeking a Senior Data Engineer to design and build data pipelines and infrastructure that transform complex, heterogeneous data into reliable resources. You will focus on the data layer underpinning analyst tools, including datasets, data models and serving layers, while shaping feature pipelines and evaluation data as the team grows.

You will establish and maintain the data platform across cloud and on-premise environments, addressing storage, compute, orchestration

Qualifications

• Experience designing, building and operating production data pipelines at scale.
• Experience with data models and curated datasets for analyst tools.
• Proficient in Python engineering, testing and CI/CD practices.
• Experience deploying data infra using infrastructure-as-code in cloud and/or on-premise environments.
• Ability to work within security constraints and audit requirements.

Responsibilities

• Design, build and operate robust data pipelines ingesting and structuring large data volumes.
• Maintain and evolve the data platform across cloud and on-premise environments.
• Collaborate with engineers, data scientists and partner teams to ensure secure, compliant data flows.

Skills

Python

CI/CD

Data pipelines

Data modeling

Infra as code

Cloud & on-prem

Security controls

Cross-team collaboration

Tools

Terraform

Kubernetes

Job description

ROLE PURPOSE

We are looking for a Senior Data Engineer to design and build the data pipelines and infrastructure that transform complex, heterogeneous data into reliable, well-structured resources. Initially, you will focus on the data layer underpinning analyst-facing tools, including curated datasets, data models, serving layers and query interfaces. As the team's machine learning capabilities grow, this foundation will also support feature pipelines, dataset versioning and evaluation data.

You will establish and maintain the team's data platform across cloud and on-premise environments, covering storage, compute, orchestration and access control. Working within Defence and partner environments, you will design solutions that meet technical and security constraints while enabling effective data use.

This is a senior individual contributor role suited to an experienced data engineer who has built and operated data systems at scale, can set technical direction, and is comfortable working within operationally constrained environments.

HOW YOU WILL MAKE AN IMPACT

Design, build and operate robust data pipelines that ingest, validate, clean and structure large volumes of varied data, ensuring quality, lineage and reproducibility from the outset. Develop the data products that support the team's tools, including data models, curated datasets, serving layers and query interfaces used by analysts and engineers. Establish and maintain the team's data platform across cloud and on-premise environments, covering storage, compute, orchestration, access control and infrastructure-as-code. Work closely with applied engineers and partner teams to understand how data will be consumed and design solutions that operate effectively within technical and security constraints. Set and maintain the team's data engineering standards, including testing, schema and contract management, documentation and secure data handling, supported by effective code review. Help shape the team's data architecture and ways of working, mentoring colleagues and advising on the future development of data and platform engineering capabilities.

ABOUT YOU

You will bring:

Significant experience designing, building and operating production data pipelines across large, complex and poorly structured datasets, including batch and streaming data. Strong Python engineering skills, including testing, packaging, code review and CI/CD practices. Experience deploying and operating data infrastructure, including storage, compute, orchestration and access control, using infrastructure-as-code in cloud and/or on-premise environments. Expertise in designing data models and developing trusted data products, such as curated datasets, serving layers and query interfaces. Experience handling data under security, access-control and audit constraints, and designing systems that respect them by default. The ability to collaborate effectively with engineers, data scientists, analysts and partner organisations, balancing technical ambition with practical delivery constraints.

Terms and Conditions

This position is offered on a fixed-term basis with a salary of £57,000-£69,000 per annum, plus an excellent benefits package, including flexible working and family-friendly policies.

Application Procedure

Please visit our jobs portal for full details of the role and application process.

Eligibility for Developed Vetting (DV) clearance is an essential requirement for this role.

EQUALITY, DIVERSITY AND INCLUSION

We value diversity of background, experience, and perspective, and are proud to be an inclusive employer. We warmly encourage applications from all backgrounds, particularly from groups currently under-represented in our sector.