Lead Data Engineer: AI Infrastructure Architect

Smart Working; Milton Keynes; Hybride; GBP 90 000 - 120 000; Plein temps

Résumé du poste

Smart Working seeks a Lead Data Engineer to build and scale the data infrastructure powering an AI assistant platform. You will shape the data architecture, pipelines, and vector databases to enable reliable real-time analytics and ML workflows.

You will collaborate with AI and backend teams to ensure the platform processes large volumes of data efficiently. This role requires four days in the office weekly in Milton Keynes and offers a path to establish the data function as the company grows.

Qualifications

• 7+ years of professional experience in dedicated data engineering roles.
• Experience designing and building data pipelines and distributed data systems.
• Experience with relational databases (PostgreSQL preferred; MySQL acceptable).
• Experience with NoSQL databases.
• Strong Python programming.
• Ability to justify architectural decisions, not just implement.
• Experience building scalable backend systems.
• Experience designing data models and storage architectures.
• Understanding of data processing performance and optimisation.
• Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch.
• Desirable: PostgreSQL, MongoDB, vector databases (Qdrant/Milvus/pgvector).
• Desirable: Python data-processing libraries (Pandas/Polars).

Responsabilités

• Architect and build scalable data pipelines and infrastructure for AI and product systems.
• Design and maintain data ingestion, transformation and storage architectures for operational and AI workloads.
• Develop and manage batch and real-time data pipelines.
• Build and optimise systems for vector search and ML data pipelines.
• Ensure data reliability, security and governance across the platform.
• Collaborate with AI and backend teams to support training, inference and features.
• Implement monitoring, observability and data quality frameworks.
• Optimize performance of large-scale datasets and queries.
• Contribute to long-term data strategy and architecture decisions.
• Act as founding data hire and help shape data culture and hiring standards.
• Partner with founders and product leadership to translate data capabilities into product decisions.

Connaissances

Data pipelines

Distributed systems

PostgreSQL

NoSQL

Python

Backend systems

Data modeling

Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Vector databases

Outils

PostgreSQL

MySQL

NoSQL databases

Python

Apache Spark

Apache Airflow

Kafka

Elasticsearch/OpenSearch

Qdrant

Milvus

pgvector

Pandas/Polars

Description du poste

Smart Working seeks a Lead Data Engineer to build and scale the data infrastructure powering an AI assistant platform. You will shape the data architecture, pipelines, and vector databases to enable reliable real-time analytics and ML workflows.

You will collaborate with AI and backend teams to ensure the platform processes large volumes of data efficiently. This role requires four days in the office weekly in Milton Keynes and offers a path to establish the data function as the company grows.