Senior Data Engineer: Spark, Databricks & AWS Expert

PRACYVA; Brussel Hoofdstad; Sur place; EUR 90 000 - 110 000; Plein temps

Résumé du poste

PRACYVA in Brussels seeks a Senior Data Engineer to design and implement data solutions across multiple sources, focusing on data quality, integration, ETL processes, and Data Lake management. The role collaborates with Data Analysts, Data Scientists, IT, and stakeholders to ensure reliable data delivery.

Key technologies include Python, PySpark, Databricks, AWS, and SQL. The ideal candidate will develop scalable ETL pipelines, optimize Spark and SQL queries, and maintain production data

Qualifications

• Strong experience with Python 3 in large-scale data platforms.
• Extensive PySpark and Databricks expertise for data processing pipelines.
• Proficiency in AWS and SQL/data warehousing design and optimization.
• Experience with data quality, data integration, ETL processes, and Data Lake management.
• Familiarity with Agile environments is a plus.

Responsabilités

• Develop and maintain data solutions using Python, PySpark, Databricks, and AWS.
• Capture and integrate structured and unstructured data from multiple sources.
• Design and implement ETL/data processing pipelines.
• Structure, standardize, map, clean, and validate data.
• Ensure data quality within the Data Lake.
• Identify and remove duplicate or invalid data.
• Develop and optimize Spark and SQL queries.
• Create and maintain Databricks ETL jobs and Spark clusters.
• Write technical specifications and documentation.
• Develop and execute unit tests.
• Participate in solution and architecture discussions.
• Establish and implement data management best practices.
• Monitor production systems and troubleshoot issues.
• Support production maintenance and continuous improvement.
• Ensure timely delivery of assigned tasks and projects.
• Collaborate with multidisciplinary teams and business stakeholders.
• Understand customer requirements and translate them into effective technical solutions.

Connaissances

Python 3

PySpark

Databricks

AWS

SQL / Relational Databases / Data Ware

Agile methodologies

Outils

Terraform

CI/CD

C#

Description du poste

PRACYVA in Brussels seeks a Senior Data Engineer to design and implement data solutions across multiple sources, focusing on data quality, integration, ETL processes, and Data Lake management. The role collaborates with Data Analysts, Data Scientists, IT, and stakeholders to ensure reliable data delivery.

Key technologies include Python, PySpark, Databricks, AWS, and SQL. The ideal candidate will develop scalable ETL pipelines, optimize Spark and SQL queries, and maintain production data