Lead Data Engineer – Equity & Real-Time AI Pipelines

Hexwired Recruitment Limited; Greater London; Hybrid; GBP 80,000 - 115,000; Full time

Benefits offered by this job

Equity options

Ownership and autonomy

Direct collaboration with founders

Competitive salary

Job summary

Hexwired Recruitment Limited is seeking a Lead Data Engineer in London (Hybrid) to design and build the data infrastructure powering an AI platform. You will own data architecture, build scalable pipelines, real-time processing, and AI-ready infra while guiding the engineering team.

You will drive data modelling, governance, and performance optimisation, collaborating with AI, backend and product teams to support model training and product development.

Qualifications

• 7+ years of experience in Data Engineering or Backend Engineering.
• Strong Python development experience.
• Experience building distributed data platforms and scalable pipelines.
• Excellent knowledge of relational and NoSQL databases.
• Experience with vector databases (Qdrant, Milvus, pgvector).
• Knowledge of Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars.
• Understanding of data modelling, storage architecture and performance optimisation.

Responsibilities

• Design and build scalable batch and real-time data pipelines.
• Develop ingestion, transformation and storage solutions for operational and AI workloads.
• Build and optimise vector search, retrieval systems and ML data pipelines.
• Ensure data reliability, governance, monitoring and observability across the platform.
• Collaborate with AI, backend and product teams for model training and inference.
• Optimise large-scale datasets and database performance.
• Help define engineering standards and mentor future data engineers.

Skills

Python

Distributed data

PostgreSQL

NoSQL

Vector databases

Spark

Airflow

Kafka

Elasticsearch

OpenSearch

Pandas

Polars

Data modelling

Storage architecture

Tools

AWS

Azure

GCP

Qdrant

Milvus

pgvector

Job description

Hexwired Recruitment Limited is seeking a Lead Data Engineer in London (Hybrid) to design and build the data infrastructure powering an AI platform. You will own data architecture, build scalable pipelines, real-time processing, and AI-ready infra while guiding the engineering team.

You will drive data modelling, governance, and performance optimisation, collaborating with AI, backend and product teams to support model training and product development.