Senior Data Engineer: AI-Ready Data Foundations

Intellias; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Intellias in London is seeking a senior Python data engineer with strong data-engineering and AI-agent experience to design and build the catalogue, semantic, entitlement and analytical layers for turning large on-premise data estates into usable assets for AI agents.

You will work on data foundations, access control, and integration across databases, time-series stores, and streaming platforms, applying LLMs to metadata and ensuring governance and security in a regulated environment.

Qualifications

• 5+ years building production data systems in Python, with strong engineering fundamentals (testing, code review, performance) and solid SQL.
• Experience building crawlers, harvesters or connector frameworks that extract inventories, schemas, field dictionaries and lineage from databases, filesystems, message platforms and API surfaces, in addition to conventional data pipelines.
• Event-driven integration with Kafka or similar, including secure producer patterns (mTLS or equivalent) and schema-managed topics.
• Experience with search and document stores that back catalogue platforms (e.g. Elasticsearch, OpenSearch, MongoDB or similar).
• Working knowledge of lineage capture and modelling, with OpenLineage or similar as a reference, and readiness to work with proprietary in-house event models.
• Experience applying LLMs to metadata work, such as drafting descriptions and classifications for human review, including quality evaluation of the generated output.
• Readiness to work inside another team's codebase, complete components that the team has designed, and contribute through its review process.
• Fluent English for written and spoken communication with client teams.

Responsibilities

• Build extraction, enrichment and registration paths that populate domain catalogues from live estates (databases, time-series stores, streaming platforms, filesystems, internal and vendor APIs) and connect them to the client's central catalogue through its existing mechanisms.
• Extend an existing scraping capability from bare dataset and symbol inventories to full metadata: descriptions, field-level dictionaries, date ranges, asset-class and cadence tags, vendor provenance.
• Load vendor schema metadata at scale, through vendor APIs, into a persistent internal knowledge base designed for step-by-step disclosure to humans and LLMs.
• Seed report and dataset inventories from existing application metadata tables and ETL sources; combine them with LLM-drafted descriptions approved by stewards.
• Integrate lineage into the client's lineage backend across batch, streaming and cross-system report chains, completing the client's existing registration designs.
• Implement the federation contract defined by the architect: stable identities, ownership, hierarchy, links and availability state exposed by each local catalogue to the central layer.
• Keep metadata current through scheduled and event-driven refresh, with explicit staleness detection and quality signals.

Skills

Python

Data engineering

SQL

Distributed systems

Kafka

LLMs

OpenLineage

Elasticsearch/OpenSearch

MongoDB

Airflow

Parquet

DataHub

Tools

Elasticsearch/OpenSearch

MongoDB

Kafka

Airflow

Parquet

DataHub

Job description

Intellias in London is seeking a senior Python data engineer with strong data-engineering and AI-agent experience to design and build the catalogue, semantic, entitlement and analytical layers for turning large on-premise data estates into usable assets for AI agents.

You will work on data foundations, access control, and integration across databases, time-series stores, and streaming platforms, applying LLMs to metadata and ensuring governance and security in a regulated environment.