Senior Data Engineer - AI-Ready Metadata & Lineage

Intellias; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Intellias is assisting a global investment management client headquartered in London. We seek a hands-on senior Python engineer with strong data-engineering skills to design and build catalogue, semantic, entitlement and analytical layers that transform large on-premise data estates into AI-ready assets for safe enterprise use.

You will lead the development of data catalogue foundations, enabling agents to access, interpret, and reason over data with quality and governance baked in.

Qualifications

• 5+ years building production data systems in Python with solid SQL and testing practices.
• Experience building crawlers, harvesters or connectors for inventories, schemas, and lineage.
• Event-driven integration with Kafka and secure producer patterns.
• Search/document stores such as Elasticsearch/OpenSearch/MongoDB.
• Knowledge of data lineage modelling with OpenLineage.
• Experience applying LLMs to metadata work.
• Fluent English in written and spoken communication.

Responsibilities

• Build extraction, enrichment and registration paths that populate domain catalogues from live estates and connect to the central catalogue.
• Extend scraping from bare dataset inventories to full metadata including descriptions and dictionaries.
• Load vendor schema metadata at scale via vendor APIs into a persistent knowledge base for humans and LLMs.
• Seed report and dataset inventories from existing metadata tables and ETL sources with LLM-drafted descriptions.
• Integrate lineage into the client's lineage backend across batch, streaming and cross-system report chains.
• Implement the federation contract with stable identities and ownership exposed by local catalogues.
• Keep metadata current through scheduled and event-driven refresh with staleness detection.

Skills

Python

SQL

Data engineering

Testing & code review

Kafka

OpenLineage

LLM metadata

English

Tools

Elasticsearch

OpenSearch

MongoDB

Kafka tooling

Job description

Intellias is assisting a global investment management client headquartered in London. We seek a hands-on senior Python engineer with strong data-engineering skills to design and build catalogue, semantic, entitlement and analytical layers that transform large on-premise data estates into AI-ready assets for safe enterprise use.

You will lead the development of data catalogue foundations, enabling agents to access, interpret, and reason over data with quality and governance baked in.