Senior Data Engineer: AI-Ready Metadata & Lineage

Intellias; City Of London; On-site; GBP 100,000 - 150,000; Full time

Job summary

Intellias is seeking a hands-on senior Python engineer with strong data-engineering skills to design and build the catalogue, semantic, entitlement and analytical layers that turn large on-premise data estates into usable AI-ready assets.

You will work on safe, scalable AI adoption across the enterprise, building extraction, enrichment and registration paths and extending existing scraping for metadata with LL M drafted descriptions and steward-approved quality controls.

Qualifications

• 5+ years building production data systems in Python with strong engineering fundamentals.
• Experience building crawlers, harvesters or connectors for inventories and metadata.
• Event-driven integration with Kafka and secure patterns (mTLS).
• Experience with search/document stores backing catalogues (Elasticsearch/OpenSearch, MongoDB).
• Knowledge of lineage capture and modelling; OpenLineage or similar reference.
• Experience applying LLMs to metadata work and quality evaluation.
• Able to work in another team's codebase and contribute via reviews.
• Fluent English for written and spoken client communication.

Responsibilities

• Build extraction, enrichment and registration paths from live estates to the central catalogue.
• Extend scraping from datasets to full metadata including descriptions and dictionaries.
• Load vendor schema metadata at scale into a persistent knowledge base.
• Seed inventories from application metadata tables and ETL sources with LL M-drafted descriptions.
• Integrate lineage into the client lineage backend across batch, streaming and reports.
• Implement federation contracts with stable identities, ownership and links.
• Keep metadata current with scheduled and event-driven refresh and quality signals.

Skills

Python

SQL

Data pipelines

Crawler frameworks

Kafka

Elasticsearch/OpenSearch

OpenLineage

LLM for metadata

English communication

Regulated environments

Tools

Elasticsearch

OpenSearch

MongoDB

Kafka

Airflow

Parquet

Job description

Intellias is seeking a hands-on senior Python engineer with strong data-engineering skills to design and build the catalogue, semantic, entitlement and analytical layers that turn large on-premise data estates into usable AI-ready assets.

You will work on safe, scalable AI adoption across the enterprise, building extraction, enrichment and registration paths and extending existing scraping for metadata with LL M drafted descriptions and steward-approved quality controls.