Data Engineer – Azure Databricks & Lakehouse Pipelines

IVC Evidensia Belgium; Belgique; Hybride; EUR 60 000 - 90 000; Plein temps

Résumé du poste

IVC Evidensia Belgium is implementing a Databricks-based data platform to support analytics, reporting and data-driven decision making. You will own the evolution of data ingestion, storage and processing, building robust pipelines and trusted datasets for business intelligence.

You will collaborate with analysts and stakeholders to ensure scalable, cost-efficient solutions, while upholding governance and best practices across the data platform.

Qualifications

• Proven experience as a Data Engineer or Analytics Engineer in a modern data environment.
• Hands-on experience with Databricks, including AI/BI Dashboards and Genie.
• Strong SQL skills and experience working with relational data.
• Strong Python and/or PySpark experience.
• Experience designing, building and maintaining ELT/ETL data pipelines.
• Experience with orchestration tools such as Azure Data Factory or equivalent.
• Good understanding of medallion architecture, including Bronze, Silver and Gold data layers.
• Experience working with cloud-based data storage and processing technologies.
• Understanding of data modelling and building consumption-ready datasets for BI and analytics.
• Experience integrating data from multiple structured and semi-structured sources, including APIs, JSON, databases and files.
• Good understanding of performance optimisation, scalability and cost management in cloud data platforms.
• Experience with Git, Azure DevOps or similar version control and delivery tooling.
• Strong problem-solving skills and ability to troubleshoot pipeline, data quality and performance issues.
• Excellent communication skills and ability to explain technical topics to non-technical stakeholders.
• Ability to work collaboratively with analysts, architects, infrastructure teams and business users.
• Excellent communication skills in Dutch and English.

Responsabilités

• Design, build, maintain and optimise data pipelines within Azure Databricks.
• Take technical ownership of data ingestion from source systems into the Belgium data platform.
• Build and maintain Bronze, Silver and Gold data layers using a medallion architecture.
• Add and integrate new data sources, including internal systems, external datasets, APIs and file-based feeds.
• Improve pipeline performance, reliability, monitoring and observability.
• Identify opportunities to speed up data processing and reduce platform running costs.
• Support the transition from legacy reporting and data infrastructure to a modern Databricks Lakehouse platform.
• Work closely with the Data Analyst to provide trusted, consumption-ready datasets for Power BI reporting and self-service analytics.
• Collaborate with business users and SMEs to understand data requirements and translate them into robust technical solutions.
• Implement data quality checks, validation rules and reconciliation processes.
• Support documentation of data flows, business definitions, assumptions and technical design decisions.
• Contribute to the definition of engineering standards, reusable patterns and platform best practices.
• Work within DevOps ways of working, using version control, deployment practices and structured delivery methods.
• Support secure development practices, including access management, scalability, performance and other non-functional requirements.
• Proactively seek innovative solutions to improve the platform, automate manual processes and increase the value delivered from data.

Connaissances

Databricks

SQL

Python

PySpark

ELT/ETL

Data modelling

APIs

Git

Azure DevOps

Performance tuning

Outils

Azure Databricks

Azure Data Factory

Git

Azure DevOps

Description du poste

IVC Evidensia Belgium is implementing a Databricks-based data platform to support analytics, reporting and data-driven decision making. You will own the evolution of data ingestion, storage and processing, building robust pipelines and trusted datasets for business intelligence.

You will collaborate with analysts and stakeholders to ensure scalable, cost-efficient solutions, while upholding governance and best practices across the data platform.