Remote Senior Data Engineer: Metadata & AI Lineage

Intellias; Valladolid; Presencial; EUR 85.000 - 110.000; Jornada completa

Descripción de la vacante

Intellias is seeking a Senior Data Engineer (Metadata & Lineage Integration) to design and build scalable data foundations in a remote-from-Spain context. You will create catalogues from live estates, extend metadata with descriptions, and enable lineage connectivity to the central catalogue, leveraging AI-ready data foundations.

You will work with Python, SQL, and modern data platforms, contributing to governance and secure data access, while coordinating with client teams in English.

Formación

• 5+ years building production data systems with strong engineering fundamentals and solid SQL.
• Experience building crawlers, harvesters, or connectors extracting inventories, schemas and lineage from databases, filesystems, and APIs.
• Event-driven integration with Kafka or similar, including secure producer patterns and schema-managed topics.
• Experience with search/document stores backing catalogues (Elasticsearch/OpenSearch/MongoDB).
• Understanding of lineage capture/modelling; OpenLineage or equivalent as reference; work with in-house event models.
• Experience applying LLMs to metadata tasks, drafting descriptions/classifications for human review.

Responsabilidades

• Build extraction, enrichment and registration paths populating domain catalogues from live estates and connect to central catalogue.
• Extend scraping capability to full metadata: descriptions, dictionaries, date ranges, asset-class tags, provenance.
• Load vendor schema metadata via APIs into internal knowledge base for human and LLM use.
• Seed inventories from metadata tables and ETL sources; combine with LLM-drafted descriptions.
• Integrate lineage into client lineage backend across batch, streaming and cross-system chains.
• Implement federation contract with stable identities, ownership, hierarchy, and availability state.

Conocimientos

Data pipeline design

OpenLineage concepts

English communication

Herramientas

Python

SQL

Kafka

Elasticsearch/OpenSearch

MongoDB

Descripción del empleo

Intellias is seeking a Senior Data Engineer (Metadata & Lineage Integration) to design and build scalable data foundations in a remote-from-Spain context. You will create catalogues from live estates, extend metadata with descriptions, and enable lineage connectivity to the central catalogue, leveraging AI-ready data foundations.

You will work with Python, SQL, and modern data platforms, contributing to governance and secure data access, while coordinating with client teams in English.