Lead Data Engineer — Shape AI Data Backbone (London, Equity)

LightWork AI; Greater London; On-site; GBP 90,000 - 140,000; Full time

Job summary

LightWork AI, based in London, is hiring a Lead Data Engineer to design and scale the data backbone of the platform. You will architect data infrastructure powering AI products, from real-time pipelines to vector databases, and work with AI and backend teams to enable reliable, intelligent data processing.

You will lead as the first senior data hire, defining data architecture, tooling, and standards while shaping the future data team and culture at LightWork AI.

Qualifications

• 7+ years in dedicated data engineering roles.
• Experience designing and building data pipelines and distributed data systems.
• Experience with relational databases (PostgreSQL preferred).
• Experience with NoSQL databases and vector databases.

Responsibilities

• Architect and build scalable data pipelines and infrastructure to support LightWork AI’s AI and product systems.
• Design and maintain data ingestion, transformation, and storage architectures for operational and AI workloads.
• Develop and manage batch and real‑time data pipelines.
• Build and optimize systems for vector search, retrieval, and ML data pipelines.
• Ensure data reliability, security, and governance across the platform.
• Collaborate with AI and backend engineering teams to support training, inference, and product features.
• Implement monitoring, observability, and data quality frameworks.
• Optimize performance of large-scale datasets and query systems.
• Contribute to technical architecture decisions and long-term data strategy.
• Act as the founding data hire, defining culture, standards and hiring bar for the data function as it scales
• Partner directly with founders and product leadership to translate data capabilities into product decisions.

Skills

Data pipelines

Distributed data systems

Python

Architectural decisions

Data governance

Performance optimization

Problem solving

Tools

Apache Spark

Apache Airflow

Kafka

Elasticsearch/OpenSearch

Job description

LightWork AI, based in London, is hiring a Lead Data Engineer to design and scale the data backbone of the platform. You will architect data infrastructure powering AI products, from real-time pipelines to vector databases, and work with AI and backend teams to enable reliable, intelligent data processing.

You will lead as the first senior data hire, defining data architecture, tooling, and standards while shaping the future data team and culture at LightWork AI.