Lead Data Engineer: AI Data Infrastructure Architect

Smart Working; Chelmsford; Hybrid; GBP 90,000 - 150,000; Full time

Benefits offered by this job

Remote-friendly

Competitive salary

Stock options

Job summary

Smart Working is seeking a Lead Data Engineer to design and scale the data infrastructure powering an AI assistant platform. You will architect real-time data pipelines, analytics infrastructure, vector databases, and ML data workflows to enable reliable, intelligent product experiences.

You will collaborate with AI and backend engineers, defining data architecture and engineering standards while shaping the future data team.

Qualifications

• 7+ years of professional data engineering experience.
• Strong experience designing and building data pipelines and distributed data systems.
• Experience with relational databases (PostgreSQL preferred).
• Experience with NoSQL databases.
• Strong Python programming experience.

Responsibilities

• Architect and build scalable data pipelines and infrastructure for AI and product systems.
• Design and maintain data ingestion, transformation and storage architectures for operational and AI workloads.
• Develop and manage batch and real-time data pipelines.
• Build and optimise vector search and ML data pipelines.
• Ensure data reliability, security and governance across the platform.
• Collaborate with AI and backend teams to support training and product features.
• Implement monitoring, observability and data quality frameworks.
• Lead the data function as it scales and define hiring standards.

Skills

Python

Data pipelines

PostgreSQL

NoSQL

Distributed systems

ML data workflows

Spark

Airflow

Kafka

Vector databases

OpenSearch

Tools

PostgreSQL

MongoDB

Elasticsearch

OpenSearch

Job description

Smart Working is seeking a Lead Data Engineer to design and scale the data infrastructure powering an AI assistant platform. You will architect real-time data pipelines, analytics infrastructure, vector databases, and ML data workflows to enable reliable, intelligent product experiences.

You will collaborate with AI and backend engineers, defining data architecture and engineering standards while shaping the future data team.