Lead Data Engineer for AI Platform & Data Infra (Remote)

Lever, Inc.; Greater London; Remote; GBP 90,000 - 150,000; Full time

Job summary

Smart Working in the United Kingdom seeks a Lead Data Engineer to design and scale the data infrastructure powering its AI assistant platform. You will own real-time data pipelines, vector databases, and ML data workflows, collaborating with AI and backend engineers to meet high-volume demands.

You will define data architecture and standards, build reliable data processing, and lead the formation of a data team as the company grows.

Qualifications

• 7+ years of professional experience in data engineering roles.
• Strong experience designing and building data pipelines and distributed data systems.
• Experience with relational databases (PostgreSQL preferred; MySQL acceptable).
• NoSQL databases experience.
• Strong Python programming experience.
• Ability to justify architectural decisions, not just implement them.
• Experience building scalable backend systems.
• Experience designing data models and storage architectures.
• Strong understanding of data processing performance and optimisation.
• Experience with Spark, Airflow, Kafka, and OpenSearch/Elasticsearch is highly desirable.
• Experience with PostgreSQL, MongoDB, and vector databases like Qdrant or Milvus is desirable.
• Experience with Pandas or Polars is desirable.

Responsibilities

• Architect and build scalable data pipelines and infrastructure for AI products.
• Design and maintain data ingestion, transformation and storage architectures for operational and AI workloads.
• Develop and manage batch and real-time data pipelines.
• Build and optimise vector search, retrieval and ML data pipelines.
• Ensure data reliability, security and governance across the platform.
• Collaborate with AI and backend teams to support training, inference and product features.
• Implement monitoring and data quality frameworks.
• Contribute to long-term data strategy and architecture decisions.

Skills

Data infrastructure

Python

Distributed systems

Data architecture

Backend systems

Data modeling

Performance optimisation

Tools

PostgreSQL

MySQL

NoSQL databases

Apache Spark

Airflow

Kafka

Elasticsearch/OpenSearch

Vector databases

Pandas/Polars

Job description

Smart Working in the United Kingdom seeks a Lead Data Engineer to design and scale the data infrastructure powering its AI assistant platform. You will own real-time data pipelines, vector databases, and ML data workflows, collaborating with AI and backend engineers to meet high-volume demands.

You will define data architecture and standards, build reliable data processing, and lead the formation of a data team as the company grows.