Senior Data Engineer: AI-Driven Data Catalog & Lineage

Intellias; Poland; On-site; PLN 260,000 - 380,000; Full time

Job summary

Intellias is seeking a hands-on senior Python data engineer to design and build data foundations for AI-enabled, on-premise environments in regulated finance. You will craft the catalogue, semantic, entitlement and analytical layers that transform large estates into agent-friendly datasets.

You will implement scalable extraction, metadata enrichment, lineage tracking and governance, working with Kafka, OpenSearch and lake formats.

Qualifications

• 5+ years building production data systems in Python.
• Experience with crawlers, harvesters or connectors to extract inventories and schemas.
• Event-driven integration with Kafka and secure patterns.
• Experience with search/document stores (Elasticsearch/OpenSearch/MongoDB).
• Experience applying LLMs to metadata and quality evaluation.
• Willing to work inside another team’s codebase and participate in reviews.
• Fluent English for written and spoken communication.

Responsibilities

• Build extraction, enrichment and registration paths that populate domain catalogues from live estates and connect to the central catalogue.
• Extend scraping to full metadata: descriptions, field dictionaries and provenance.
• Load vendor schema metadata via vendor APIs into the internal knowledge base.
• Seed inventories from application metadata tables and ETL sources with LLm-drafted descriptions.
• Integrate lineage into the client’s lineage backend across batch, streaming and cross-system chains.
• Implement federation contract with stable identities, ownership, and availability state.
• Keep metadata current through scheduled refresh and quality signals.

Skills

Python

Data Engineering

SQL

Kafka

OpenLineage

LLM integration

Tools

Elasticsearch/OpenSearch

MongoDB

Parquet

Airflow

Job description

Intellias is seeking a hands-on senior Python data engineer to design and build data foundations for AI-enabled, on-premise environments in regulated finance. You will craft the catalogue, semantic, entitlement and analytical layers that transform large estates into agent-friendly datasets.

You will implement scalable extraction, metadata enrichment, lineage tracking and governance, working with Kafka, OpenSearch and lake formats.