Lead Data Engineer

Understanding Recruitment; Greater London; On-site; GBP 90,000 - 150,000; Full time

Benefits offered by this job

equity

Job summary

Understanding Recruitment is seeking a lead data engineer to own the data architecture behind an AI product at an early stage. You’ll shape real-time data pipelines, vector search, and analytics, with the freedom to make core technical decisions and set standards.

You’ll architect scalable batch and real-time data pipelines, design ingestion/storage, and build infrastructure for vector search and ML workflows, mentoring the data team as the platform evolves.

Qualifications

• Experience designing scalable pipelines and distributed data systems.
• Experience with relational and NoSQL databases.
• Hands-on experience with vector databases.
• Good understanding of data modelling, performance and storage architecture.
• Experience with Spark, Airflow, Kafka or Elasticsearch/OpenSearch.

Responsibilities

• Architect scalable batch and real-time data pipelines.
• Design ingestion, transformation and storage systems.
• Build infrastructure for vector search, retrieval and ML workflows.
• Improve data quality, observability and reliability.
• Optimize large-scale datasets and query performance.
• Support AI training, inference and product features.
• Make long-term architecture and tooling decisions.
• Help grow and mentor the data engineering function.

Skills

Data architecture

Distributed systems

Data modeling

Relational databases

NoSQL databases

Vector databases

Observability

Education

Bachelor's degree in Computer Science or related field

Tools

Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Vector databases

Job description

Want to own the data architecture behind an AI product while the foundations are still being built?

This is an early-stage lead role where you’ll shape the systems powering real-time data, AI workflows, vector search and analytics — with the freedom to make the key technical decisions.

What’s in it for you?

• Own the data architecture and technical direction
• Build the data backbone of an AI-first product from the ground up
• Work closely with AI, backend and product engineers
• Solve problems across streaming, vector search and ML data infrastructure
• Help define tooling, standards and the future data team
• Competitive salary + equity
• Join early enough to have genuine influence over how things are built

What you’ll be doing

• Architecting scalable batch and real-time data pipelines
• Designing ingestion, transformation and storage systems
• Building infrastructure for vector search, retrieval and ML workflows
• Improving data quality, observability and reliability
• Optimising large-scale datasets and query performance
• Supporting AI training, inference and product features
• Making long-term architecture and tooling decisions
• Helping grow and mentor the data engineering function

What you’ll bring

• Experience designing scalable pipelines and distributed data systems
• Experience with relational and NoSQL databases
• Hands‑on experience with vector databases
• Good understanding of data modelling, performance and storage architecture
• Experience with technologies such as Spark, Airflow, Kafka or Elasticsearch/OpenSearch

Experience with AI/ML platforms, cloud infrastructure or event-driven architectures would be useful, but you don’t need everything on the list.