Senior Data Architect

idpp; Basingstoke; On-site; GBP 90,000 - 150,000; Full time

Job summary

idpp is seeking a Principal Data & AI Engineer in the United Kingdom to own the data and AI engineering function and shape product direction. You will lead the design of scalable data platforms, collaborate with researchers and software teams, and drive governance, security, and reliability across enterprise-grade systems.

You will mentor engineers, drive technical strategy, and translate complex business challenges into scalable AI-enabled solutions, all within highly secure environments and

Qualifications

• Significant commercial experience in data/AI engineering, leadership, and platform ownership.
• Experience building AI-enabled platforms with LLMs, embeddings, vector databases and RAG architectures.
• Strong Python and SQL skills with modern software engineering principles.
• Knowledge of cloud-native architectures, containers, and orchestrations in restricted environments.
• Experience deploying AI platforms in regulated sectors and MLOps tooling.

Responsibilities

• Lead architecture and development of scalable AI and data platforms for ML and LLM workloads.
• Design data pipelines from diverse secure sources and ensure governance and observability.
• Build secure, containerised platforms using Docker/Kubernetes for restricted environments.
• Mentor engineers and drive technical strategy across AI teams.
• Collaborate with stakeholders to translate business challenges into scalable solutions.

Skills

Python

SQL

LLMs

Vector databases

RAG architectures

Distributed data processing

Cloud-native

MLOps

Model lifecycle management

Technical leadership

Communication

Security in regulated environments

Education

Degree in Computer Science / Software Engineering / Data Engineering or related

Tools

Docker

Kubernetes

Azure

AWS

Terraform

Helm

Ansible

Spark

Kafka

Airflow

Dagster

Job description

We're partnering with an innovative UK technology company developing a next-generation AI platform for critical defence and healthcare applications. As they continue to scale, they're looking for a Principal Data & AI Engineer to provide technical leadership across the design, architecture, and delivery of enterprise-grade AI and data platforms.

This is a rare opportunity to take ownership of the data and AI engineering function within a rapidly growing business, shaping the technical direction of products that solve complex, real-world problems. You'll work alongside AI researchers, software engineers, and product teams to build highly secure, scalable data platforms that power advanced AI capabilities in mission-critical environments.

If you're passionate about modern data engineering, large-scale AI systems, and influencing technical strategy while remaining hands-on, this role offers genuine autonomy and long-term career progression.

What You'll Be Doing

• Lead the architecture and development of scalable AI and data platforms supporting advanced machine learning and LLM workloads.
• Design and build high-performance data pipelines capable of processing structured and unstructured data from multiple secure sources.
• Architect robust data ingestion, transformation and orchestration frameworks using modern cloud-native tooling.
• Build and optimise Retrieval-Augmented Generation (RAG) pipelines and enterprise knowledge retrieval platforms.
• Develop and optimise vector search capabilities using technologies such as Weaviate, Pinecone, Milvus or Elasticsearch.
• Engineer highly available APIs and backend services using Python (FastAPI/Flask) and modern software engineering practices.
• Design secure, containerised platforms using Docker and Kubernetes for deployment into restricted environments.
• Drive best practice around data modelling, governance, observability, testing and platform reliability.
• Mentor and provide technical leadership to engineers across the wider software and AI teams.
• Work closely with stakeholders to translate complex business challenges into scalable technical solutions.

What We're Looking For

• Significant commercial experience as a Senior, Lead or Principal Data Engineer, AI Engineer or Platform Engineer.
• Strong expertise in Python and SQL, alongside modern software engineering principles.
• Deep understanding of distributed data processing and orchestration technologies (e.g. Spark, Kafka, Airflow, Dagster or similar).
• Experience building AI-enabled platforms using Large Language Models (LLMs), embeddings, vector databases and RAG architectures.
• Hands-on experience with vector databases such as Pinecone, Weaviate, Milvus or Elasticsearch.
• Strong understanding of cloud-native architecture, Docker and Kubernetes.
• Experience designing resilient microservice architectures and RESTful APIs.
• Knowledge of data governance, security, metadata management and production data operations.
• Comfortable leading technical direction, mentoring engineers and influencing architecture decisions.
• Excellent communication skills with the ability to engage technical and non-technical stakeholders.
• Degree in Computer Science, Software Engineering, Data Engineering or a related discipline (or equivalent commercial experience).
• Experience deploying AI platforms into secure or highly regulated environments.
• Knowledge of MLOps tooling and model lifecycle management.
• Experience with Azure, AWS or private cloud infrastructure.
• Familiarity with Infrastructure as Code (Terraform, Helm, Ansible or similar).
• Experience within Defence, National Security, Healthcare, Government or other regulated industries.
• Exposure to knowledge graphs, semantic search or graph databases.
• Experience leading technical strategy within a fast-growing technology business.

Security Requirements

Due to the nature of the work, applicants must be eligible for SC Security Clearance