Lead Data Engineer - Remote-First AI Data Platform

Smart Working; Greater London; Remote; GBP 110,000 - 160,000; Full time

Job summary

Smart Working is seeking a Lead Data Engineer to architect and scale data infrastructure powering an AI assistant for property management platforms. You will own real-time and batch pipelines, vector databases, and ML data workflows, working with AI and backend teams to ensure reliable and efficient data processing.

As the founding data hire, you will shape the data architecture, technology stack and hiring bar for the function, collaborating with founders and product leadership to translate

Qualifications

• 7+ years of professional data engineering experience.
• Experience designing and building data pipelines and distributed systems.
• Relational DBs with PostgreSQL preferred; MySQL acceptable.
• Experience with NoSQL databases.
• Experience with vector databases used in AI.
• Strong Python programming experience.
• Familiar with Spark, Airflow, Kafka, Elasticsearch or OpenSearch.
• Experience with PostgreSQL, MongoDB and vector DBs like Qdrant/Milvus/pgvector.
• Pandas or Polars for data processing.

Responsibilities

• Architect and build scalable data pipelines and infrastructure.
• Design ingestion, transformation and storage architectures.
• Develop batch and real-time data pipelines.
• Build and optimise systems for vector search, retrieval and ML data pipelines.
• Ensure data reliability, security and governance across the platform.
• Collaborate with AI and backend engineering teams to support training and inference.
• Implement monitoring, observability and data quality frameworks.
• Optimise the performance of large-scale datasets and query systems.
• Contribute to technical architecture decisions and long-term data strategy.
• Act as the founding data hire, defining culture, standards and the hiring bar for the data function as it scales.
• Partner directly with founders and product leadership to translate data capabilities into product decisions.

Skills

Python

PostgreSQL

NoSQL

Vector databases

Data pipelines

Distributed systems

Spark

Airflow

Kafka

Elasticsearch/OpenSearch

MongoDB

Pandas/Polars

Tools

PostgreSQL tooling

pgvector

Qdrant

Milvus

MongoDB tooling

Elasticsearch/OpenSearch tooling

Job description

Smart Working is seeking a Lead Data Engineer to architect and scale data infrastructure powering an AI assistant for property management platforms. You will own real-time and batch pipelines, vector databases, and ML data workflows, working with AI and backend teams to ensure reliable and efficient data processing.

As the founding data hire, you will shape the data architecture, technology stack and hiring bar for the function, collaborating with founders and product leadership to translate