Senior Databricks Data Engineer

MPower Plus; Zaventem; On-site; EUR 90,000 - 150,000; Full time

Job summary

MPower Plus is seeking an experienced Senior Databricks Data Engineer in Belgium to design scalable data pipelines on the Databricks Lakehouse platform. You will leverage Databricks, SQL, Lakeflow Jobs, PySpark notebooks, Python, and Unity Catalog to deliver reliable data solutions.

You will design, develop, and optimize data pipelines, orchestrate workflow jobs, manage Unity Catalog tables, and implement end-to-end ETL/ELT processes while ensuring data quality and performance.

Qualifications

• Minimum 4+ years as a Data Engineer with strong Databricks expertise.
• 10+ years in the data domain, especially data warehousing and ETL.
• Proficient in PySpark, Spark SQL, and Python programming.
• Git version control experience (commit/check-out).
• Knowledge of Lakeflow Jobs and Unity Catalog is essential.

Responsibilities

• Design, develop, and optimize data pipelines using Databricks, Python, PySpark, and Spark SQL.
• Orchestrate and schedule workflow jobs that invoke notebooks and Python packages.
• Work with Unity Catalog, manage tables, and streaming jobs.
• Implement data ingestion, transformation, and loading into the Databricks Lakehouse.
• Build and maintain reliable ETL/ELT processes ensuring data quality.
• Optimize Databricks workloads for performance and cost efficiency.
• Troubleshoot data issues, ensuring data integrity and accuracy.
• Stay current with Databricks features, best practices, and trends.

Skills

Databricks

Data Engineering

Lakeflow

Unity Catalogue

Python

PySpark

ETL

Job description

Mandatory Skills: Databricks, Data Engineering, Lakeflow, Unity Catalogue, Python, PySpark, ETL.

About the Role:

We are looking for an experienced Senior Databricks Data Engineer with strong expertise in Databricks, SQL, Lakeflow Jobs, PySpark notebooks, Python, and Unity Catalogue. This role requires hands-on experience in designing scalable data pipelines and delivering reliable data solutions on the Databricks Lakehouse platform.

Key Responsibilities:

• Design, develop, and optimize data pipelines using Databricks, Python, PySpark, and Spark SQL.
• Orchestrate and schedule workflow jobs that invoke notebooks, Python wheel packages, and Python notebooks.
• Work with Unity Catalog, manage tables, and streaming jobs.
• Implement and manage data ingestion, transformation, and loading processes from various sources into the Databricks Lakehouse.
• Build and maintain efficient and reliable ETL/ELT processes to ensure data quality and availability.
• Optimise Databricks workloads for performance, cost efficiency, and scalability.
• Troubleshoot and resolve data-related issues while ensuring data integrity and accuracy.
• Stay current with the latest Databricks features, best practices, and industry trends.

Required skills & Qualifications:

• At least 4 years of proven experience as a Data Engineer with a strong focus on Databricks (mandatory).
• Strong knowledge of Databricks architecture, Lakeflow Jobs, and data modelling.
• At least 10 years of experience in the data domain, particularly in data warehousing and ETL (mandatory).
• Expertise in PySpark, Spark SQL, and Python programming (mandatory).
• Hands-on experience with Git, including code check-in and check-out processes (mandatory).
• Strong understanding of data warehousing, data lakes, and Lakehouse architecture.
• Databricks Certified Data Engineer Associate or Professional certification is an advantage.

Soft Skills:

• Strong communication and stakeholder collaboration skills.
• Proactive, self-driven, and accountable mindset.
• Excellent analytical and problem-solving abilities.
• Ability to work effectively in Agile and cross-functional teams.
• Customer-focused with a commitment to quality delivery.
• Ability to converse seamlessly in English and one of either Dutch or French or both