Hybrid Lead Data Engineer - AI Infra & Pipelines

Smart Working; Woking; Hybrid; GBP 90,000 - 150,000; Full time

Job summary

Smart Working is seeking a Lead Data Engineer to design and lead our data infrastructure powering an AI assistant platform. You'll architect real-time data pipelines, analytics infrastructure, vector databases, and ML data workflows, partnering with AI and backend teams to handle large volumes of data.

You will define data architecture, tooling, and standards, and help shape the future data team while working in a hybrid setup that requires four days in the office per week.

Qualifications

• 7+ years of professional data engineering experience.
• Experience designing data pipelines and distributed systems.
• Relational DBs (PostgreSQL preferred) and NoSQL.
• Strong Python programming.
• Experience with Spark, Airflow, Kafka, Elasticsearch/OpenSearch is desirable.
• Experience with vector databases (Qdrant, Milvus, pgvector) is desirable.

Responsibilities

• Architect and build scalable data pipelines and infrastructure to support AI and product systems.
• Design and maintain data ingestion, transformation and storage architectures for operational and AI workloads.
• Develop and manage batch and real-time data pipelines.
• Build and optimise systems for vector search, retrieval and machine learning data pipelines.
• Ensure data reliability, security and governance across the platform.
• Collaborate with AI and backend engineering teams to support training, inference and product features.
• Implement monitoring, observability and data quality frameworks.
• Optimise the performance of large-scale datasets and query systems.
• Contribute to technical architecture decisions and long-term data strategy.
• Act as the founding data hire, defining culture, standards and the hiring bar for the data function as it scales.
• Partner directly with founders and product leadership to translate data capabilities into product decisions.

Skills

Data pipelines

Distributed data systems

Python

Architectural decisions

Backend systems

Data modeling

Performance optimization

Data governance

Data security

Tools

PostgreSQL

MySQL

NoSQL

Apache Spark

Apache Airflow

Kafka

Elasticsearch/OpenSearch

Vector databases

Qdrant

Milvus

pgvector

Pandas/Polars

Job description

Smart Working is seeking a Lead Data Engineer to design and lead our data infrastructure powering an AI assistant platform. You'll architect real-time data pipelines, analytics infrastructure, vector databases, and ML data workflows, partnering with AI and backend teams to handle large volumes of data.

You will define data architecture, tooling, and standards, and help shape the future data team while working in a hybrid setup that requires four days in the office per week.