Lead Data Engineer

Hexwired Recruitment Limited; Greater London; Hybrid; GBP 80,000 - 115,000; Full time

Benefits offered by this job

Equity options

Job summary

Hexwired Recruitment Limited is seeking a senior Data/Backend Engineer in London to design and implement scalable batch and real-time data pipelines. You will work on ingestion, transformation, and storage for operational and AI workloads, including vector search and ML data flows.

You’ll collaborate with AI, backend and product teams to optimize data reliability, governance, and platform performance, while mentoring junior engineers. Equity options are available in a fast-growing AI startup.

Qualifications

• 7+ years of experience in Data/Backend Engineering.
• Strong Python development experience.
• Experience building distributed data platforms and scalable pipelines.
• Excellent knowledge of PostgreSQL and NoSQL databases.

Responsibilities

• Design and build scalable batch and real-time data pipelines.
• Develop data ingestion, transformation, and storage solutions for operational and AI workloads.
• Build and optimise vector search, retrieval systems, and ML data pipelines.
• Ensure data reliability, governance, monitoring, and observability across the platform.
• Work closely with AI, backend, and product teams to support model training, inference, and product development.
• Optimise large-scale datasets and database performance.
• Help define engineering standards and mentor future data engineers.

Skills

Python

Data engineering

Distributed data platforms

PostgreSQL

NoSQL

Vector databases

Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Pandas

Polars

AWS

Azure

GCP

JavaScript/Node.js

Event-driven architecture

AI/ML platforms

Tools

OpenSearch

pgvector

Qdrant

Milvus

Job description

Salary: £80,000 - 115,000 per year

Requirements

• 7+ years of experience in Data Engineering or Backend Engineering.
• Strong Python development experience.
• Experience building distributed data platforms and scalable pipelines.
• Excellent knowledge of PostgreSQL or similar relational databases, and NoSQL databases.
• Experience with vector databases such as Qdrant, Milvus, or pgvector.
• Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas, and Polars.
• Strong understanding of data modelling, storage architecture, and performance optimisation.
• Desirable: experience working on AI or Machine Learning platforms.
• Desirable: knowledge of cloud platforms such as AWS, Azure, or GCP.
• Desirable: JavaScript/Node.js experience.
• Desirable: event-driven architecture and stream processing experience.
• Desirable: previous experience within a high-growth start-up environment.

Responsibilities

• Design and build scalable batch and real-time data pipelines.
• Develop data ingestion, transformation, and storage solutions for operational and AI workloads.
• Build and optimise vector search, retrieval systems, and ML data pipelines.
• Ensure data reliability, governance, monitoring, and observability across the platform.
• Work closely with AI, backend, and product teams to support model training, inference, and product development.
• Optimise large-scale datasets and database performance.
• Help define engineering standards and mentor future data engineers.

Technologies

• AI
• Airflow
• AWS
• OpenSearch
• Azure
• Backend
• Cloud
• ElasticSearch
• GCP
• Support
• JavaScript
• Kafka
• Machine Learning
• NoSQL
• PostgreSQL
• Python
• Spark
• pandas
• NodeJS
• C#
• Embedded
• FPGA
• Golang
• Linux

More

We are an exciting AI start-up developing an intelligent automation platform that is transforming the B2C service. This is a hybrid role based in London with a salary of £80,000 to £115,000. As one of our first senior data hires, you will have the opportunity to shape our data architecture, work directly alongside our founders and senior engineers, and enjoy a high level of ownership, autonomy, and technical influence. We offer competitive salary with equity options and the chance to build technology that is transforming an entire industry.