Senior Data Engineer - AI-Ready Metadata & Lineage

Intellias; Slough; On-site; GBP 90,000 - 120,000; Full time

Job summary

Intellias is seeking a hands-on senior Python data engineer to design and build the catalogue, semantic, entitlement and analytical layers that transform large on-premise data estates for AI agents. You will craft extraction, enrichment and registration paths, extend scraping to full metadata, and load vendor schema metadata into a central knowledge base.

You will work with event-driven architectures (Kafka), search stores (Elasticsearch/OpenSearch), and OpenLineage-like lineage models, applying

Qualifications

• 5+ years building production data systems in Python with strong engineering fundamentals and solid SQL.
• Experience building crawlers, harvesters or connectors extracting inventories, schemas and lineage from databases, filesystems, messages and APIs.
• Event-driven integration with Kafka or similar, including secure producer patterns and schema-managed topics.
• Experience with search/document stores backing catalogue platforms (Elasticsearch, OpenSearch, MongoDB).
• Knowledge of lineage modelling with OpenLineage and readiness to work with in-house event models.
• Experience applying LLMs to metadata work, drafting descriptions and classifications for human review.
• Ability to work inside another team's codebase, complete components and contribute via reviews.
• Fluent English for written and spoken communication with client teams.

Responsibilities

• Build extraction, enrichment and registration paths populating domain catalogues from live estates (databases, time-series stores, streaming platforms, filesystems, internal and vendor APIs).
• Extend existing scraping from bare datasets to full metadata: descriptions, field dictionaries, date ranges, asset-class and cadence tags, vendor provenance.
• Load vendor schema metadata at scale via vendor APIs into a persistent internal knowledge base for human and LLM use.
• Seed report and dataset inventories from application metadata tables and ETL sources; combine with LLM-drafted descriptions approved by stewards.
• Integrate lineage into the client's lineage backend across batch, streaming and cross-system report chains.
• Implement the federation contract defined by the architect: stable identities, ownership, hierarchy, links and availability state exposed by each local catalogue.
• Keep metadata current through scheduled and event-driven refresh with staleness detection and quality signals.

Skills

Python programming

Data engineering

SQL

Kafka

Data catalog & lineage

Elasticsearch / OpenSearch

LLM integration

English communication

Education

Bachelor's degree in Computer Science or related field

Tools

Kafka

Elasticsearch/OpenSearch

MongoDB

Airflow

DataHub

OpenLineage

Parquet

Job description

Intellias is seeking a hands-on senior Python data engineer to design and build the catalogue, semantic, entitlement and analytical layers that transform large on-premise data estates for AI agents. You will craft extraction, enrichment and registration paths, extend scraping to full metadata, and load vendor schema metadata into a central knowledge base.

You will work with event-driven architectures (Kafka), search stores (Elasticsearch/OpenSearch), and OpenLineage-like lineage models, applying