Senior Data Engineer - Metadata & Lineage in AI Catalog

Intellias; Greater London; On-site; GBP 46,000 - 50,000; Full time

Job summary

Intellias, a London-headquartered technology partner, is looking for a hands-on Python data engineer to build and evolve production data estates enabling AI-enabled discovery and catalogues.

You will design and implement extraction, enrichment and registration paths across databases, time-series stores, and APIs, connecting them to our central catalogue. Extend metadata with descriptions and dictionaries and collaborate to deliver scalable data foundations for AI agents.

Qualifications

• 5+ years building production data systems in Python with strong engineering fundamentals.
• Experience building crawlers, harvesters or connector frameworks for inventories, schemas and lineage.
• Event-driven integration with Kafka or similar and secure patterns.
• Experience with search and document stores (Elasticsearch/OpenSearch, MongoDB).
• Familiarity with data lineage and OpenLineage concepts.
• Experience applying LLMs to metadata work and quality evaluation.

Responsibilities

• Build extraction, enrichment and registration paths to populate domain catalogues from live estates and connect to central catalogue.
• Extend scraping capabilities to full metadata with descriptions and dictionaries.
• Load vendor schema metadata at scale into internal knowledge base for human and LLM use.
• Seed inventories from application metadata and ETL sources, with LLM-drafted descriptions.
• Integrate lineage into lineage backend across batch and streaming systems.
• Implement federation contracts with stable identities and data ownership.
• Keep metadata current via scheduled and event-driven refresh with quality signals.

Skills

Python

5+ years experience

SQL

Testing

Code review

Performance

Fluent English

Crawlers / harvesters

AI agents usage

Tools

Kafka

OpenSearch

MongoDB

Airflow

Parquet

DataHub

MS SQL Server

Job description

Intellias, a London-headquartered technology partner, is looking for a hands-on Python data engineer to build and evolve production data estates enabling AI-enabled discovery and catalogues.

You will design and implement extraction, enrichment and registration paths across databases, time-series stores, and APIs, connecting them to our central catalogue. Extend metadata with descriptions and dictionaries and collaborate to deliver scalable data foundations for AI agents.