Senior Data Engineer

PRACYVA; Brussel; Sur place; EUR 90 000 - 130 000; Plein temps

Résumé du poste

PRACYVA in Brussels is seeking a Senior Data Engineer to design and implement end-to-end data pipelines using Python, PySpark, Databricks, AWS, and SQL. You will collaborate with Data Analysts, Data Scientists, IT teams, and business stakeholders to deliver reliable data products in a scalable, enterprise environment.

You will own ETL/data processing pipelines, Data Lake management, data quality, and production stability, optimize Spark and SQL queries, create Databricks ETL jobs and Spark

Qualifications

• Strong experience with Python 3 and PySpark.
• Extensive Databricks and AWS expertise.
• Proficient SQL and data warehousing skills.
• Experience in building ETL pipelines and data lakes.
• Familiarity with Agile methodologies.

Responsabilités

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Connaissances

Python 3

PySpark

Databricks

AWS

SQL / Relational Databases / Data Ware

Outils

Apache Airflow

Azure DevOps

Terraform

Postman

Insomnia

Description du poste

Role Overview

We are looking for a Senior Data Engineer with strong experience in Python, PySpark, Databricks, AWS, and SQL to join the Downstream Data & IT team in Brussels.

The consultant will be responsible for developing, integrating, processing, and maintaining data solutions across multiple data sources and platforms. The role will focus on ensuring data quality, data integration, ETL processes, Data Lake management, and production stability.

The candidate will work closely with Data Analysts, Data Scientists, IT teams, business stakeholders, and the Chief Data Officer to ensure reliable and high-quality data delivery.

Key Responsibilities

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Technical Skills

Mandatory

• Python 3 – Strong experience
• PySpark – Strong experience
• Databricks – Strong experience
• AWS – Strong experience
• SQL / Relational Databases / Data Warehousing – Strong experience

Good to Have

• Apache Airflow
• Azure DevOps / TFS / VSTS
• Terraform
• Postman / Insomnia
• CI/CD
• C#

Databricks Experience

Candidates should have experience with:

• Creating and executing ETL tasks
• Managing Spark clusters
• Developing data pipelines
• Data transformation and processing
• SQL query development and optimization
• Data Lake environments
• Performance tuning

Functional Skills

• Strong understanding of relational databases, SQL, and data warehouses.
• Experience working with Databricks data engineering platforms.
• Ability to collaborate with technical and business teams.
• Strong analytical and problem-solving skills.
• Ability to work independently and proactively.
• Good understanding of data management and data quality.
• Ability to understand business requirements and deliver suitable technical solutions.
• Experience working in Agile environments is a plus.

Nice to Have – Domain Experience

Experience in any of the following will be an advantage:

• Energy / Utilities industry
• Energy trading
• Portfolio management
• Risk management
• Energy forecasting
• Gas and power markets

Key Responsibilities in the Data Management Area

The consultant will help ensure the quality and usability of enterprise data by:

• Collecting data from different applications and external sources.
• Integrating data from multiple systems.
• Structuring and standardizing data.
• Mapping available data elements.
• Cleaning and removing duplicate data.
• Validating incoming data.
• Supporting the creation and maintenance of data repositories.
• Ensuring reliable data availability for Data Analysts and Data Scientists.

Soft Skills & Behavioral Competencies

The candidate should demonstrate:

• Strong analytical and problem-solving abilities.
• Excellent communication and interpersonal skills.
• Ability to work with multiple stakeholders.
• Strong attention to detail and quality.
• Ability to work in a fast-paced, multidisciplinary environment.
• Ability to handle confidential and sensitive customer data.
• Ability to simplify and abstract complex technical topics.
• Creative and entrepreneurial approach to problem-solving.
• Strong ownership and accountability.
• High level of integrity and ethical standards.

Deliverables

The consultant will be expected to provide:

• Monthly time tracking
• Activity reports
• KPI dashboards
• Workload estimations
• Technical and delivery documentation
• Project schedules
• Production monitoring and maintenance reports

10+ years of Data Engineering experience with strong hands-on expertise in

:Python + PySpark + Databricks + AWS + SQ

LThe ideal candidate should have strong experience building ETL/data pipelines, managing Databricks/Spark environments, optimizing SQL, ensuring data quality, and supporting production systems in an enterprise environment