Lead Data Engineer — AI Platform & Real-Time Data

Hexwired Recruitment Limited; City Of London; Hybrid; GBP 80,000 - 115,000; Full time

Benefits offered by this job

Equity options

Shape data architecture

Autonomy and influence

Job summary

Hexwired Recruitment Limited is partnering with an AI start-up to hire a Lead Data Engineer in London. You will own data architecture, build scalable pipelines for AI workloads, and shape the engineering team for rapid growth.

You will design real-time and batch data systems, optimise vector search pipelines, and collaborate with AI, backend and product teams to support model training and production. Hybrid work in London with strong ownership and technical influence.

Qualifications

• 7+ years of data engineering or backend experience.
• Strong Python development experience.
• Experience building distributed data pipelines and platforms.
• Excellent knowledge of PostgreSQL and NoSQL databases.
• Experience with vector databases (Qdrant, Milvus, pgvector).
• Familiarity with Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas, Polars.

Responsibilities

• Design and build scalable batch and real-time data pipelines.
• Develop ingestion, transformation and storage solutions for AI workloads.
• Build vector search and ML data pipelines.
• Ensure data reliability, governance, and observability across the platform.
• Collaborate with AI/backend/product teams for model training and deployment.
• Optimize large-scale datasets and database performance.
• Define engineering standards and mentor engineers.

Skills

5+ years experience

Python

Distributed data platforms

PostgreSQL

NoSQL databases

Vector databases

Apache Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Pandas

Polars

Data modelling

Job description

Hexwired Recruitment Limited is partnering with an AI start-up to hire a Lead Data Engineer in London. You will own data architecture, build scalable pipelines for AI workloads, and shape the engineering team for rapid growth.

You will design real-time and batch data systems, optimise vector search pipelines, and collaborate with AI, backend and product teams to support model training and production. Hybrid work in London with strong ownership and technical influence.