Senior Data Engineer Consultant

Keystone Solutions; Brussel Hoofdstad; Sur place; EUR 70 000 - 90 000; Plein temps

Résumé du poste

Keystone Solutions is seeking a Senior Data Engineer for a consultancy mission at a client site to build an on-premise Data Exchange Platform.

You will design data ingestion pipelines, implement Bronze-Silver-Gold transformations, manage Delta Lake tables, and ensure governance, security, and data sovereignty in a multi-domain environment.

Qualifications

• Experience designing and deploying data ingestion pipelines.

Responsabilités

• Design and deploy data ingestion pipelines from files, APIs, relational databases.

Connaissances

Dagster

Data acquisition

Data architecture

Data engineering

Data Governance

DataHub

datalake

dbt-core

Kafka

Metadata management

POWER BI

Python/Pandas/Spark

SPARK

SQL

Outils

Dagster

Airflow

OpenMetadata

DataHub

Delta Lake

dbt-core

DuckDB

Power BI

Python

Spark

SQL

Description du poste

Mission Overview:

Keystone Solutions is seeking a Senior Data Engineer for a consultancy mission at a client site. You will be part of a dynamic team working on the Data Exchange Platform (DEP) project, which involves building an on-premise data platform for a governmental organization. This role emphasizes the importance of security, governance, and data sovereignty.

Main Responsibilities:

• Design and deploy data ingestion pipelines from heterogeneous sources: files, APIs, relational databases.
• Implement pipelines for incremental and streaming ingestion.
• Ensure reliability, idempotence, and traceability of each pipeline.
• Develop transformation models on Bronze, Silver, and Gold layers (Medallion architecture).
• Implement best practices for data historization (e.g., SCD2, upsert, append only).
• Develop exports to Delta Lake.
• Manage and maintain Delta Lake tables.
• Guarantee transactional consistency of data in a multi-domain environment.
• Implement and maintain orchestration assets.
• Monitor runs, manage errors, and automate alerts.
• Push technical and business metadata to the data platform.
• Maintain end-to-end lineage.
• Contribute to the drafting of DUA, SLA, and associated governance documents.
• Develop endpoints to expose the Gold layer to BI dashboards.
• Collaborate with reporting teams for BI dashboard feeding.

Required Technical Skills:

• Proficiency in Python.
• Proficiency in SQL.
• Proven experience with dbt-core (models, snapshots, macros, tests, profiles).
• Good knowledge of DuckDB as an embedded analytical query engine.
• Experience with Delta Lake (ACID transactions, time travel, OPTIMIZE/VACUUM).
• Knowledge of relational databases: MSSQL, PostgreSQL (advanced SQL, CDC).
• Experience with a data orchestrator: Dagster, Airflow, or equivalent.
• Comfortable with Linux/OpenShift and using PowerShell (Windows dev environment).
• Familiarity with governance tools: DataHub, OpenMetadata, or equivalent.
• Experience in BI development.

Preferred Skills:

• Experience with Kafka/Debezium for real-time data capture.
• Knowledge of Spark (Spark SQL, Thrift Server, Beeline).
• Experience with Power BI Report Server (PBIRS) or Apache Superset.
• Awareness of data security in restricted access environments.
• Knowledge of Lakehouse concepts (Medallion architecture), DWH, and Datalake.

Soft Skills:

• Rigorous and autonomous in managing complex multi-domain pipelines.
• Ability to document technical decisions and simplify for non-technical stakeholders.
• Team spirit.
• Curious and solution-oriented mindset.

What We Offer:

• A stable mission that is long-term.
• A high-impact project in a secure and technically stimulating environment.
• A modern and open-source stack: DLT, dbt, DuckDB, Delta Lake, Dagster.
• An experienced and supportive team focused on quality and best practices.
• Opportunities for skill enhancement in governance, orchestration, and lakehouse.

If you are ready to tackle technical and strategic challenges in a dynamic consultancy environment, apply today at Keystone Solutions Career Portal.

Duration:

As soon as possible - 31/12/2026 5 months (full time)

Skills required:

• Dagster - Level: Junior - Most recent: Any time
• Data acquisition (ETL, ELT, ...) - Level: Confirmed - Most recent: Any time
• Data architecture - Level: Junior - Most recent: Any time
• data engineering - Level: Confirmed - Most recent: Any time
• Data Governance - Level: Confirmed - Most recent: Any time
• DataHub - Level: Junior - Most recent: Any time
• datalake - Level: Junior - Most recent: Any time
• dbt - Level: Junior - Most recent: Any time
• Kafka - Level: Junior - Most recent: Any time
• Metadata management - Level: Confirmed - Most recent: Any time
• POWER BI - Level: Junior - Most recent: Any time
• Python (from a Data Engineer perspective), Pandas & Apache Spark - Level: Confirmed - Most recent: Any time
• SPARK - Level: Junior - Most recent: Any time
• SQL - Level: Confirmed - Most recent: Any time

Language requirements:

Dutch
Level Active knowledge
English
Level Active knowledge
French
Level Native