Lead Data Architect, Real-Time AI Pipelines & Vector Search

Understanding Recruitment; Greater London; On-site; GBP 90,000 - 150,000; Full time

Benefits offered by this job

equity

Job summary

Understanding Recruitment is seeking a lead data engineer to own the data architecture behind an AI product at an early stage. You’ll shape real-time data pipelines, vector search, and analytics, with the freedom to make core technical decisions and set standards.

You’ll architect scalable batch and real-time data pipelines, design ingestion/storage, and build infrastructure for vector search and ML workflows, mentoring the data team as the platform evolves.

Qualifications

• Experience designing scalable pipelines and distributed data systems.
• Experience with relational and NoSQL databases.
• Hands-on experience with vector databases.
• Good understanding of data modelling, performance and storage architecture.
• Experience with Spark, Airflow, Kafka or Elasticsearch/OpenSearch.

Responsibilities

• Architect scalable batch and real-time data pipelines.
• Design ingestion, transformation and storage systems.
• Build infrastructure for vector search, retrieval and ML workflows.
• Improve data quality, observability and reliability.
• Optimize large-scale datasets and query performance.
• Support AI training, inference and product features.
• Make long-term architecture and tooling decisions.
• Help grow and mentor the data engineering function.

Skills

Data architecture

Distributed systems

Data modeling

Relational databases

NoSQL databases

Vector databases

Observability

Education

Bachelor's degree in Computer Science or related field

Tools

Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Vector databases

Job description

Understanding Recruitment is seeking a lead data engineer to own the data architecture behind an AI product at an early stage. You’ll shape real-time data pipelines, vector search, and analytics, with the freedom to make core technical decisions and set standards.

You’ll architect scalable batch and real-time data pipelines, design ingestion/storage, and build infrastructure for vector search and ML workflows, mentoring the data team as the platform evolves.