Lead Data Engineer - Real-Time Pipelines & AI Infra

Hexwired Recruitment Limited; City Of London; Hybrid; GBP 80,000 - 115,000; Full time

Benefits offered by this job

Equity options

Job summary

Hexwired Recruitment Limited is seeking a Lead Data Engineer in London (hybrid) to own the data architecture, build scalable pipelines, and enable AI-ready infrastructure for a high-growth AI platform. You will lead data infrastructure, real-time processing, and ML data workflows, shaping the engineering team and standards.

You will collaborate with AI, backend and product teams, optimise data performance, and drive governance and observability across the platform, with equity options and strong

Qualifications

• 7+ years in Data or Backend Engineering.
• Strong Python development.
• Experience building distributed data platforms.
• Expertise in PostgreSQL and NoSQL databases.
• Experience with vector databases (Qdrant, Milvus, pgvector).
• Knowledge of Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas/Polars.
• Good data modelling and storage architecture skills.

Responsibilities

• Design and build scalable batch and real-time data pipelines.
• Develop data ingestion, transformation and storage solutions for operational and AI workloads.
• Build and optimise vector search, retrieval systems and ML data pipelines.
• Ensure data reliability, governance, monitoring and observability across the platform.
• Work closely with AI, backend and product teams to support model training, inference and product development.
• Optimise large-scale datasets and database performance.
• Help define engineering standards and mentor future data engineers.

Skills

Python

Data pipelines

Data modelling

PostgreSQL

NoSQL

Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Pandas/Polars

Tools

Qdrant

Milvus

pgvector

Apache Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Pandas/Polars

Job description

Hexwired Recruitment Limited is seeking a Lead Data Engineer in London (hybrid) to own the data architecture, build scalable pipelines, and enable AI-ready infrastructure for a high-growth AI platform. You will lead data infrastructure, real-time processing, and ML data workflows, shaping the engineering team and standards.

You will collaborate with AI, backend and product teams, optimise data performance, and drive governance and observability across the platform, with equity options and strong