Remote Lead Data Engineer for AI Platform & Scalable Infra

Smart Working; United Kingdom; Remote; GBP 90,000 - 130,000; Full time

Job summary

Smart Working is seeking a Lead Data Engineer to build and scale the data infrastructure powering an AI assistant platform. You will define data architecture, oversee real-time pipelines, vector databases, and ML data workflows, collaborating with AI and product teams to ensure reliable processing of large data volumes.

You will shape engineering standards, establish governance, and hire the data function as it scales, partnering with founders to translate data capabilities into product

Qualifications

• 7+ years in dedicated data engineering roles.
• Experience with relational and NoSQL databases.
• Strong Python programming experience.
• Proven ability to design scalable data pipelines and architectures.
• Experience with Spark, Airflow, Kafka, Elasticsearch/OpenSearch.
• Experience with vector databases like Qdrant, Milvus or pgvector.

Responsibilities

• Architect and build scalable data pipelines and infrastructure.
• Design ingestion, transformation and storage architectures for AI workloads.
• Develop batch and real-time data pipelines.
• Build systems for vector search and ML data pipelines.
• Ensure data reliability, security and governance across the platform.
• Collaborate with AI and backend teams to support training and inference.
• Implement monitoring, observability and data quality frameworks.
• Contribute to long-term data strategy and architecture decisions.

Skills

Data engineering

Distributed systems

Python

PostgreSQL

NoSQL

Data pipelines

Data modelling

Architectural decisions

Cloud (GCP/AWS/Azure)

Open-source tools

Tools

Apache Spark

Apache Airflow

Kafka

Elasticsearch or OpenSearch

pgvector

Qdrant or Milvus

Pandas/Polars

Job description

Smart Working is seeking a Lead Data Engineer to build and scale the data infrastructure powering an AI assistant platform. You will define data architecture, oversee real-time pipelines, vector databases, and ML data workflows, collaborating with AI and product teams to ensure reliable processing of large data volumes.

You will shape engineering standards, establish governance, and hire the data function as it scales, partnering with founders to translate data capabilities into product