Senior Data Engineer

PRACYVA; Brussel Hoofdstad; Sur place; EUR 90 000 - 110 000; Plein temps

Résumé du poste

PRACYVA in Brussels seeks a Senior Data Engineer to design and implement data solutions across multiple sources, focusing on data quality, integration, ETL processes, and Data Lake management. The role collaborates with Data Analysts, Data Scientists, IT, and stakeholders to ensure reliable data delivery.

Key technologies include Python, PySpark, Databricks, AWS, and SQL. The ideal candidate will develop scalable ETL pipelines, optimize Spark and SQL queries, and maintain production data

Qualifications

• Strong experience with Python 3 in large-scale data platforms.
• Extensive PySpark and Databricks expertise for data processing pipelines.
• Proficiency in AWS and SQL/data warehousing design and optimization.
• Experience with data quality, data integration, ETL processes, and Data Lake management.
• Familiarity with Agile environments is a plus.

Responsabilités

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Connaissances

Python 3

PySpark

Databricks

AWS

SQL / Relational Databases / Data Ware

Agile methodologies

Outils

Terraform

CI/CD

C#

Description du poste

We are looking for a Senior Data Engineer with strong experience in Python, PySpark, Databricks, AWS, and SQL to join the Downstream Data & IT team in Brussels.

The consultant will be responsible for developing, integrating, processing, and maintaining data solutions across multiple data sources and platforms. The role will focus on ensuring data quality, data integration, ETL processes, Data Lake management, and production stability.

The candidate will work closely with Data Analysts, Data Scientists, IT teams, business stakeholders, and the Chief Data Officer to ensure reliable and high-quality data delivery.

Key Responsibilities

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Technical Skills

Mandatory

• Python 3 – Strong experience
• PySpark – Strong experience
• Databricks – Strong experience
• AWS – Strong experience
• SQL / Relational Databases / Data Warehousing – Strong experience

Good to Have

• Terraform
• CI/CD
• C#

Databricks Experience

Candidates should have experience with:

• Creating and executing ETL tasks
• Data transformation and processing
• SQL query development and optimization
• Data Lake environments
• Performance tuning

Functional Skills

• Strong understanding of relational databases, SQL, and data warehouses.
• Experience working with Databricks data engineering platforms.
• Ability to collaborate with technical and business teams.
• Strong analytical and problem-solving skills.
• Ability to work independently and proactively.
• Good understanding of data management and data quality.
• Ability to understand business requirements and deliver suitable technical solutions.
• Experience working in Agile environments is a plus.

Nice to Have – Domain Experience

Experience in any of the following will be an advantage:

• Energy / Utilities industry
• Energy trading
• Energy forecasting
• Gas and power markets

Key Responsibilities in the Data Management Area

The consultant will help ensure the quality and usability of enterprise data by:

• Collecting data from different applications and external sources.
• Integrating data from multiple systems.
• Structuring and standardizing data.
• Cleaning and removing duplicate data.
• Supporting the creation and maintenance of data repositories.
• Ensuring reliable data availability for Data Analysts and Data Scientists.

The candidate should demonstrate:

• Strong analytical and problem-solving abilities.
• Excellent communication and interpersonal skills.
• Ability to work with multiple stakeholders.
• Strong attention to detail and quality.
• Ability to work in a fast-paced, multidisciplinary environment.
• Ability to handle confidential and sensitive customer data.
• Ability to simplify and abstract complex technical topics.
• Creative and entrepreneurial approach to problem-solving.
• Strong ownership and accountability.
• High level of integrity and ethical standards.

The consultant will be expected to provide:

• Monthly time tracking
• Activity reportsTechnical and delivery documentation
• Production monitoring and maintenance reports

10+ years of Data Engineering experience with strong hands-on expertise in

:Python + PySpark + Databricks + AWS + SQ

LThe ideal candidate should have strong experience building ETL/data pipelines, managing Databricks/Spark environments, optimizing SQL, ensuring data quality, and supporting production systems in an enterprise environment

.