Senior Data Engineer: AI-Ready Metadata & Lineage

Intellias; Chantada; Presencial; EUR 65.000 - 90.000; Jornada completa

Descripción de la vacante

Intellias in Spain seeks a Senior Data Engineer specializing in Metadata & Lineage Integration to design and build catalogue, semantic, and analytical layers for AI-ready data estates. The role focuses on creating scalable data foundations and governance across databases, file systems, and streaming platforms while leveraging AI agents.

The candidate will implement open standards such as OpenLineage and collaborate across teams to ensure secure, scalable access to data for AI applications.

Formación

• 5+ years building production data systems in Python with strong engineering fundamentals (testing, code review, performance) and solid SQL.
• Experience building crawlers, harvesters or connector frameworks to extract inventories, schemas and lineage from databases, filesystems and APIs.
• Event-driven integration with Kafka or similar, including secure producer patterns and schema-managed topics.
• Experience with search/document stores backing catalogue platforms (Elasticsearch/OpenSearch/MongoDB).
• Working knowledge of lineage capture/modelling (OpenLineage or similar) and readiness to work with in-house event models.
• Experience applying LLMs to metadata tasks, including drafting descriptions/classifications for human review.

Responsabilidades

• Build extraction, enrichment and registration paths populating domain catalogues from live estates and connect to the central catalogue.
• Extend scraping to full metadata: descriptions, field dictionaries, date ranges, tags, and vendor provenance.
• Load vendor schema metadata at scale via vendor APIs into a persistent knowledge base for human/LLM access.
• Seed inventories from application metadata and ETL sources; combine with LLM-drafted descriptions reviewed by stewards.
• Integrate lineage into the client’s lineage backend across batch, streaming and cross-system chains.
• Implement the federation contract: stable identities, ownership, hierarchy and links for each local catalogue to the central layer.
• Keep metadata current through scheduled and event-driven refresh with staleness/quality signals.

Conocimientos

Python

SQL

Testing

Code review

Performance optimization

Event-driven integration

Kafka

OpenLineage

Herramientas

Kafka

Elasticsearch/OpenSearch/MongoDB

Airflow

OpenLineage

DataHub

Descripción del empleo

Intellias in Spain seeks a Senior Data Engineer specializing in Metadata & Lineage Integration to design and build catalogue, semantic, and analytical layers for AI-ready data estates. The role focuses on creating scalable data foundations and governance across databases, file systems, and streaming platforms while leveraging AI agents.

The candidate will implement open standards such as OpenLineage and collaborate across teams to ensure secure, scalable access to data for AI applications.