Senior Data Engineer (Metadata & Lineage)

Intellias; Greater London; On-site; GBP 46,000 - 50,000; Full time

Job summary

Intellias, a London-headquartered technology partner, is looking for a hands-on Python data engineer to build and evolve production data estates enabling AI-enabled discovery and catalogues.

You will design and implement extraction, enrichment and registration paths across databases, time-series stores, and APIs, connecting them to our central catalogue. Extend metadata with descriptions and dictionaries and collaborate to deliver scalable data foundations for AI agents.

Qualifications

• 5+ years building production data systems in Python with strong engineering fundamentals.
• Experience building crawlers, harvesters or connector frameworks for inventories, schemas and lineage.
• Event-driven integration with Kafka or similar and secure patterns.
• Experience with search and document stores (Elasticsearch/OpenSearch, MongoDB).
• Familiarity with data lineage and OpenLineage concepts.
• Experience applying LLMs to metadata work and quality evaluation.

Responsibilities

• Build extraction, enrichment and registration paths to populate domain catalogues from live estates and connect to central catalogue.
• Extend scraping capabilities to full metadata with descriptions and dictionaries.
• Load vendor schema metadata at scale into internal knowledge base for human and LLM use.
• Seed inventories from application metadata and ETL sources, with LLM-drafted descriptions.
• Integrate lineage into lineage backend across batch and streaming systems.
• Implement federation contracts with stable identities and data ownership.
• Keep metadata current via scheduled and event-driven refresh with quality signals.

Skills

Python

5+ years experience

SQL

Testing

Code review

Performance

Fluent English

Crawlers / harvesters

AI agents usage

Tools

Kafka

OpenSearch

MongoDB

Airflow

Parquet

DataHub

MS SQL Server

Job description

Salary: £46,000 - 50,000 per year

Requirements:

• 5+ years building production data systems in Python, with strong engineering fundamentals (testing, code review, performance) and solid SQL.
• Experience building crawlers, harvesters or connector frameworks that extract inventories, schemas, field dictionaries and lineage from databases, filesystems, message platforms and API surfaces, in addition to conventional data pipelines.
• Event-driven integration with Kafka or similar, including secure producer patterns (mTLS or equivalent) and schema-managed topics.
• Experience with search and document stores that back catalogue platforms (e.g. Elasticsearch, OpenSearch, MongoDB or similar).
• Working knowledge of lineage capture and modelling, with OpenLineage or similar as a reference, and readiness to work with proprietary in-house event models.
• Experience applying LLMs to metadata work, such as drafting descriptions and classifications for human review, including quality evaluation of the generated output.
• Readiness to work inside another teams codebase, complete components that the team has designed, and contribute through its review process.
• Fluent English for written and spoken communication with client teams.
• Time-series and tick stores (e.g. kdb+ or similar columnar time-series databases), market-data vendor schema APIs, symbology and asset-class concepts (market-data opening).
• MS SQL Server estates, reporting and BI systems, inventory extraction from application metadata tables (reporting opening).
• Columnar and lake formats (Parquet or similar), large object stores, orchestration platforms (Airflow or similar).
• Working-level knowledge of graph databases; data contracts and data quality frameworks; catalogue platforms (DataHub or similar) on the ingestion side.
• Day-to-day use of AI coding agents; building data services consumed by AI agents.
• Experience in financial services or other regulated on-premise environments.

Responsibilities:

• Build extraction, enrichment and registration paths that populate domain catalogues from live estates (databases, time-series stores, streaming platforms, filesystems, internal and vendor APIs) and connect them to our central catalogue through its existing mechanisms.
• Extend an existing scraping capability from bare dataset and symbol inventories to full metadata: descriptions, field-level dictionaries, date ranges, asset-class and cadence tags, vendor provenance.
• Load vendor schema metadata at scale, through vendor APIs, into a persistent internal knowledge base designed for step-by-step disclosure to humans and LLMs.
• Seed report and dataset inventories from existing application metadata tables and ETL sources; combine them with LLM-drafted descriptions approved by stewards.
• Integrate lineage into our lineage backend across batch, streaming and cross-system report chains, completing our existing registration designs.
• Implement the federation contract defined by the architect: stable identities, ownership, hierarchy, links and availability state exposed by each local catalogue to the central layer.
• Keep metadata current through scheduled and event-driven refresh, with explicit staleness detection and quality signals.

Technologies:

• AI
• AI Agents
• Airflow
• API
• OpenSearch
• Architect
• Backend
• ETL
• ElasticSearch
• Exposed
• Kafka
• LLM
• Machine Learning
• MongoDB
• Python
• SQL
• Security

More:

Our client is a leading global investment management company headquartered in London. It manages over $228 billion in assets and serves institutional investors, pension funds, wealth managers, and other sophisticated clients worldwide. The firm specializes in quantitative investing, alternative investments, systematic trading strategies, and technology-driven asset management. Data science, machine learning, and AI are core components of its investment and research processes. As part of our collaboration, we are focusing on two foundational capabilities required to enable safe and scalable AI adoption across the enterprise: Agentic Security and AI-Ready Data Foundations. We build the data foundations that make AI useful and safe inside regulated financial firms. The value of AI is capped by the data its agents can reach: if an agent cannot find, interpret, trace or be correctly permissioned against data, the capability is useless, or worse, unsafe. This is a hands‑on senior role for an excellent Python engineer with strong data‑engineering skills who is genuinely comfortable building with AI agents. You will design and build the catalogue, semantic, entitlement and analytical layers that turn large on‑premise data estates into something agents can use.

last updated 38 week of 2026