Data Engineer

Intellias; Poland; On-site; PLN 260,000 - 380,000; Full time

Job summary

Intellias is seeking a hands-on senior Python data engineer to design and build data foundations for AI-enabled, on-premise environments in regulated finance. You will craft the catalogue, semantic, entitlement and analytical layers that transform large estates into agent-friendly datasets.

You will implement scalable extraction, metadata enrichment, lineage tracking and governance, working with Kafka, OpenSearch and lake formats.

Qualifications

• 5+ years building production data systems in Python.
• Experience with crawlers, harvesters or connectors to extract inventories and schemas.
• Event-driven integration with Kafka and secure patterns.
• Experience with search/document stores (Elasticsearch/OpenSearch/MongoDB).
• Experience applying LLMs to metadata and quality evaluation.
• Willing to work inside another team’s codebase and participate in reviews.
• Fluent English for written and spoken communication.

Responsibilities

• Build extraction, enrichment and registration paths that populate domain catalogues from live estates and connect to the central catalogue.
• Extend scraping to full metadata: descriptions, field dictionaries and provenance.
• Load vendor schema metadata via vendor APIs into the internal knowledge base.
• Seed inventories from application metadata tables and ETL sources with LLm-drafted descriptions.
• Integrate lineage into the client’s lineage backend across batch, streaming and cross-system chains.
• Implement federation contract with stable identities, ownership, and availability state.
• Keep metadata current through scheduled refresh and quality signals.

Skills

Python

Data Engineering

SQL

Kafka

OpenLineage

LLM integration

Tools

Elasticsearch/OpenSearch

MongoDB

Parquet

Airflow

Job description

We build the data foundations that make AI useful and safe inside regulated financial firms. The value of AI is capped by the data its agents can reach: if an agent cannot find, interpret, trace or be correctly permissioned against data, the capability is useless, or worse, unsafe. Your job is to close that gap.

This is a hands-on senior role for an excellent Python engineer with strong data-engineering skills who is genuinely comfortable building with AI agents. You will design and build the catalogue, semantic, entitlement and analytical layers that turn large on-premise data estates into something agents can use.

What you will do

• Build extraction, enrichment and registration paths that populate domain catalogues from live estates (databases, time-series stores, streaming platforms, filesystems, internal and vendor APIs) and connect them to the client’s central catalogue through its existing mechanisms.
• Extend an existing scraping capability from bare dataset and symbol inventories to full metadata: descriptions, field-level dictionaries, date ranges, asset-class and cadence tags, vendor provenance.
• Load vendor schema metadata at scale, through vendor APIs, into a persistent internal knowledge base designed for step-by-step disclosure to humans and LLMs.
• Seed report and dataset inventories from existing application metadata tables and ETL sources; combine them with LLM-drafted descriptions approved by stewards.
• Integrate lineage into the client’s lineage backend across batch, streaming and cross-system report chains, completing the client’s existing registration designs.
• Implement the federation contract defined by the architect: stable identities, ownership, hierarchy, links and availability state exposed by each local catalogue to the central layer.
• Keep metadata current through scheduled and event-driven refresh, with explicit staleness detection and quality signals.

What you need for this

• 5+ years building production data systems in Python, with strong engineering fundamentals (testing, code review, performance) and solid SQL.
• Experience building crawlers, harvesters or connector frameworks that extract inventories, schemas, field dictionaries and lineage from databases, filesystems, message platforms and API surfaces, in addition to conventional data pipelines.
• Event-driven integration with Kafka or similar, including secure producer patterns (mTLS or equivalent) and schema-managed topics.
• Experience with search and document stores that back catalogue platforms (e.g. Elasticsearch, OpenSearch, MongoDB or similar).
• Working knowledge of lineage capture and modelling, with OpenLineage or similar as a reference, and readiness to work with proprietary in-house event models.
• Experience applying LLMs to metadata work, such as drafting descriptions and classifications for human review, including quality evaluation of the generated output.
• Readiness to work inside another team’s codebase, complete components that the team has designed, and contribute through its review process.
• Fluent English for written and spoken communication with client teams.

Will be a plus

• Time-series and tick stores (e.g. kdb+ or similar columnar time-series databases), market-data vendor schema APIs, symbology and asset-class concepts (market-data opening).
• MS SQL Server estates, reporting and BI systems, inventory extraction from application metadata tables (reporting opening).
• Columnar and lake formats (Parquet or similar), large object stores, orchestration platforms (Airflow or similar).
• Working-level knowledge of graph databases; data contracts and data quality frameworks; catalogue platforms (DataHub or similar) on the ingestion side.
• Day-to-day use of AI coding agents; building data services consumed by AI agents.
• Experience in financial services or other regulated on-premise environments.

What it’s like to work at Intellias

At Intellias, where technology takes center stage, people always come before processes. By creating a comfortable atmosphere in our team, we empower individuals to unlock their true potential and achieve extraordinary results. That’s why we offer a range of benefits that support your well-being and charge your professional growth.

We are committed to fostering equity, diversity, and inclusion as an equal opportunity employer. All applicants will be considered for employment without discrimination based based on race, color, religion, age, gender, nationality, disability, sexual orientation, gender identity or expression, veteran status, or any other characteristic protected by applicable law.

We welcome and celebrate the uniqueness of every individual. Join Intellias for a career where your perspectives and contributions are vital to our shared success.